Oct 30 20:26:03.705675 ip-10-0-139-195 systemd[1]: Starting Kubernetes Kubelet... Oct 30 20:26:04.100200 ip-10-0-139-195 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:26:04.101182 ip-10-0-139-195 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 20:26:04.101182 ip-10-0-139-195 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:26:04.101182 ip-10-0-139-195 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:26:04.105720 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105570 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:26:04.105720 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105718 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:26:04.105720 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105722 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:26:04.105720 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105726 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105731 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105736 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105739 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105743 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105745 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105748 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105750 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105753 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105756 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105758 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105761 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105763 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105766 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105769 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105771 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105774 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105776 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105779 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:26:04.105888 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105781 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105784 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105786 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105789 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105801 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105804 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105807 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105809 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105812 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105815 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105817 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105819 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105822 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105825 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105827 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105830 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105834 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105836 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105839 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105841 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:26:04.106319 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105844 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105847 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105849 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105852 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105854 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105857 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105859 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105862 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105864 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105867 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105870 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105872 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105875 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105877 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105880 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105882 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105884 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105887 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105890 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105893 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:26:04.106804 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105896 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105899 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105901 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105904 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105906 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105908 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105911 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105914 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105917 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105920 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105922 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105924 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105927 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105929 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105932 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105934 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105937 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105939 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105943 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:26:04.108039 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105948 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105951 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105955 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105958 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105961 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105963 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105966 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105968 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105971 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105973 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105976 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105979 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105987 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105992 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105996 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.105998 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106001 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106004 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106006 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:26:04.108503 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106009 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106012 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106014 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106017 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106021 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106023 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106836 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106843 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106846 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106849 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106852 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106855 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106857 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106860 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106863 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106866 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106869 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106872 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106875 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106877 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:26:04.108941 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106880 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106883 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106887 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106889 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106892 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106895 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106898 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106900 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106903 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106906 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106908 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106912 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106914 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106917 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106919 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106922 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106924 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106927 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106931 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:26:04.109426 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106935 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106938 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106942 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106944 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106947 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106949 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106952 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106954 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106957 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106960 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106963 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106966 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106969 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106972 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106974 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106976 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106979 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106982 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106984 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:26:04.109886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106987 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106989 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106992 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106995 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.106997 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107000 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107002 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107008 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107010 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107013 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107016 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107018 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107020 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107023 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107026 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107028 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107031 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107033 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107036 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107038 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:26:04.110339 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107041 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107043 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107046 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107049 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107052 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107054 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107057 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107060 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107062 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107065 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107067 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107070 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107072 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107074 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107077 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107079 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107082 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107085 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107087 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107091 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:26:04.110844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107093 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107096 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107098 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107100 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107103 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107105 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107107 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107110 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107112 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107114 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107117 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107119 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107122 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107124 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107640 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107646 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107650 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107653 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107656 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:26:04.111346 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107658 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107661 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107664 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107667 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107669 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107672 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107674 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107677 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107679 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107682 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107685 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107687 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107691 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107694 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107696 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107699 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107701 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107704 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107707 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107709 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:26:04.111814 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107711 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107714 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107716 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107719 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107721 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107724 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107726 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107730 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107734 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107737 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107740 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107743 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107745 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107748 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107750 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107753 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107755 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107758 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107760 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:26:04.112275 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107763 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107765 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107767 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107770 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107772 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107776 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107778 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107780 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107783 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107785 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107788 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107790 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107793 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107795 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107798 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107801 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107803 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107806 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107808 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107810 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:26:04.112880 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107813 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107816 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107818 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107820 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107823 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107826 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107828 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107831 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107833 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107835 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107838 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107840 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107843 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107845 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107848 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107850 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107853 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107856 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107858 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:26:04.113359 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107861 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107863 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107866 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107868 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107871 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107873 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107876 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107880 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107883 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107886 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107889 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107891 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107895 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107899 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107902 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107905 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107907 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107909 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107912 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:26:04.113819 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107915 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107918 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107921 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.107924 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108912 2572 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108922 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108929 2572 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108933 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108938 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108941 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108945 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108950 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108953 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108956 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108960 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108963 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108966 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108969 2572 flags.go:64] FLAG: --cgroup-root="" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108971 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108974 2572 flags.go:64] FLAG: --client-ca-file="" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108979 2572 flags.go:64] FLAG: --cloud-provider="external" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108982 2572 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108985 2572 flags.go:64] FLAG: --cluster-domain="" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108988 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 20:26:04.114301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108991 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108994 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.108997 2572 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109001 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109004 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109007 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109010 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109014 2572 flags.go:64] FLAG: --contention-profiling="false" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109017 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109020 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109023 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109026 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109030 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109033 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109036 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109039 2572 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109042 2572 flags.go:64] FLAG: --enable-server="true" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109044 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109048 2572 flags.go:64] FLAG: --event-burst="100" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109052 2572 flags.go:64] FLAG: --event-qps="50" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109055 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109058 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109061 2572 flags.go:64] FLAG: --eviction-hard="" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109065 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 20:26:04.114921 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109068 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109071 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109074 2572 flags.go:64] FLAG: --eviction-soft="" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109077 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109083 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109086 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109089 2572 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109092 2572 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109095 2572 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109098 2572 flags.go:64] FLAG: --feature-gates="" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109102 2572 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109105 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109108 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109111 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109114 2572 flags.go:64] FLAG: --healthz-port="10248" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109117 2572 flags.go:64] FLAG: --help="false" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109120 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109123 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109126 2572 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109130 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109133 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109136 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109139 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 20:26:04.115631 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109141 2572 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109144 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109147 2572 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109150 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109153 2572 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109156 2572 flags.go:64] FLAG: --kube-reserved="" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109158 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109161 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109164 2572 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109167 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109170 2572 flags.go:64] FLAG: --lock-file="" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109172 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109175 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109179 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109185 2572 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109188 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109191 2572 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109194 2572 flags.go:64] FLAG: --logging-format="text" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109197 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109201 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109203 2572 flags.go:64] FLAG: --manifest-url="" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109206 2572 flags.go:64] FLAG: --manifest-url-header="" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109210 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109213 2572 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109217 2572 flags.go:64] FLAG: --max-pods="110" Oct 30 20:26:04.116603 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109220 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109223 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109227 2572 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109230 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109233 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109236 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109239 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109246 2572 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109249 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109252 2572 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109255 2572 flags.go:64] FLAG: --pod-cidr="" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109258 2572 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109260 2572 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109263 2572 flags.go:64] FLAG: --pods-per-core="0" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109266 2572 flags.go:64] FLAG: --port="10250" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109269 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109272 2572 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-050d8981ac5e8b88f" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109275 2572 flags.go:64] FLAG: --qos-reserved="" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109278 2572 flags.go:64] FLAG: --read-only-port="10255" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109281 2572 flags.go:64] FLAG: --register-node="true" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109285 2572 flags.go:64] FLAG: --register-with-taints="" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109291 2572 flags.go:64] FLAG: --registry-burst="10" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109294 2572 flags.go:64] FLAG: --registry-qps="5" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109297 2572 flags.go:64] FLAG: --reserved-cpus="" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109300 2572 flags.go:64] FLAG: --reserved-memory="" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109304 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 20:26:04.117664 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109307 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109310 2572 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109313 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109315 2572 flags.go:64] FLAG: --runonce="false" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109318 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109321 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109325 2572 flags.go:64] FLAG: --seccomp-default="false" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109327 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109331 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109335 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109338 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109341 2572 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109344 2572 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109347 2572 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109350 2572 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109353 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109356 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109359 2572 flags.go:64] FLAG: --system-cgroups="" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109361 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109366 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109369 2572 flags.go:64] FLAG: --tls-cert-file="" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109372 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109375 2572 flags.go:64] FLAG: --tls-min-version="" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109378 2572 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109381 2572 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 20:26:04.118544 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109395 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109399 2572 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109402 2572 flags.go:64] FLAG: --v="2" Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109406 2572 flags.go:64] FLAG: --version="false" Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109410 2572 flags.go:64] FLAG: --vmodule="" Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109414 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109417 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109550 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109554 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109557 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109559 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109562 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109564 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109567 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109570 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109575 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109577 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109581 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109584 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109586 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109589 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:26:04.119514 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109592 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109594 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109598 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109602 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109605 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109608 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109610 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109613 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109616 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109619 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109621 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109624 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109628 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109630 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109633 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109636 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109638 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109641 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109643 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:26:04.120436 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109646 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109648 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109651 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109653 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109656 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109659 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109661 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109665 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109668 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109670 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109673 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109676 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109679 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109681 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109684 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109686 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109689 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109691 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109694 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109696 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:26:04.121137 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109699 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109701 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109704 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109706 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109709 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109712 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109716 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109719 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109722 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109725 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109728 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109730 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109733 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109736 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109738 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109741 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109743 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109746 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109749 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109752 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:26:04.121787 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109755 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109757 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109760 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109763 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109766 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109768 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109771 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109773 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109776 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109778 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109781 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109783 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109786 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109788 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109791 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109793 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109796 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109799 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109802 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:26:04.122379 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109804 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:26:04.122872 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109807 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:26:04.122872 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109810 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:26:04.122872 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109812 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:26:04.122872 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109815 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:26:04.122872 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109818 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:26:04.122872 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109820 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:26:04.122872 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109822 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:26:04.122872 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109825 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:26:04.122872 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109828 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:26:04.122872 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109830 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:26:04.122872 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109833 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:26:04.122872 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109836 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:26:04.122872 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109839 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:26:04.122872 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.109864 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:26:04.122872 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109905 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:26:04.122872 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109910 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109914 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109917 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109919 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109922 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109925 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109927 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109930 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109932 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109935 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109938 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109940 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109942 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109945 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109947 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109950 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109953 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109956 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109958 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109961 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:26:04.123548 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109963 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109966 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109968 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109971 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109973 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109976 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109978 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109981 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109985 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109988 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109990 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109994 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.109997 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110000 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110003 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110006 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110008 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110011 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110013 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:26:04.124128 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110016 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110018 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110021 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110023 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110026 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110028 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110031 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110033 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110036 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110038 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110040 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110043 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110045 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110048 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110050 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110053 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110056 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110058 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110061 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110063 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:26:04.124712 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110066 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110069 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110072 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110074 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110077 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110079 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110082 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110085 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110088 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110090 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110093 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110096 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110098 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110101 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110103 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110105 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110108 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110110 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110113 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:26:04.125184 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110115 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110118 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110120 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110123 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110125 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110127 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110130 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110132 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110135 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110137 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110139 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110142 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110144 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110146 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110150 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110153 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110156 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110160 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110164 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:26:04.125772 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110167 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110169 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110172 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110175 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110178 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110181 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110183 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.110186 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.118158 2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.118178 2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.118782 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.118795 2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118897 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118906 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118911 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118915 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118920 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118924 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118929 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:26:04.126309 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118933 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118938 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118942 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118946 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118953 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118960 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118966 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118971 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118976 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118981 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118986 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118991 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.118996 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119001 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119005 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119010 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119015 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119019 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119024 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:26:04.126775 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119028 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119032 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119036 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119050 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119055 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119059 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119063 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119068 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119072 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119076 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119080 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119084 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119089 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119093 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119097 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119102 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119106 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119110 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119114 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119118 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:26:04.127523 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119123 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119127 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119131 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119136 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119141 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119147 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119154 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119159 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119164 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119169 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119173 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119177 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119182 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119187 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119191 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119195 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119201 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119206 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119210 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:26:04.128310 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119214 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119219 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119223 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119228 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119232 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119236 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119240 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119244 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119248 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119253 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119257 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119261 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119265 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119270 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119274 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119279 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119283 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119286 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119290 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119295 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:26:04.129230 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119299 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119304 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119309 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119313 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119317 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119322 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119326 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119330 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119334 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119338 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119343 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119347 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119351 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119356 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119360 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119364 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119369 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119373 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119377 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119398 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:26:04.129931 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119402 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:26:04.130692 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.119445 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:26:04.130692 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119676 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:26:04.130692 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119684 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:26:04.130692 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119690 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:26:04.130692 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119694 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:26:04.130692 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119698 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:26:04.130692 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119702 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:26:04.130692 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119706 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:26:04.130692 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119711 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:26:04.130692 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119715 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:26:04.130692 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119719 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:26:04.130692 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119724 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:26:04.130692 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119728 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:26:04.130692 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119732 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:26:04.130692 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119737 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:26:04.130692 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119741 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119745 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119749 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119754 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119758 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119762 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119767 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119775 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119781 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119787 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119793 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119797 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119803 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119808 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119813 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119817 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119821 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119826 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119830 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119834 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:26:04.131217 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119838 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119842 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119846 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119851 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119855 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119859 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119864 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119868 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119872 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119876 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119880 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119884 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119889 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119896 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119902 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119907 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119912 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119917 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119921 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:26:04.131750 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119925 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119930 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119934 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119939 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119943 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119947 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119951 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119955 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119959 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119963 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119967 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119972 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119976 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119981 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119986 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119990 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119994 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.119998 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120003 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:26:04.132201 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120007 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120011 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120015 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120019 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120023 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120027 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120031 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120035 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120039 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120044 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120048 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120052 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120056 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120061 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120065 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120070 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120074 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120078 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120082 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:26:04.132845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120087 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:26:04.133289 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120091 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:26:04.133289 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120095 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:26:04.133289 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120099 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:26:04.133289 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120103 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:26:04.133289 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120107 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:26:04.133289 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120112 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:26:04.133289 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120116 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:26:04.133289 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120124 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:26:04.133289 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120128 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:26:04.133289 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120132 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:26:04.133289 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120136 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:26:04.133289 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120141 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:26:04.133289 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120145 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:26:04.133289 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120149 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:26:04.133289 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.120189 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:26:04.133289 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120242 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120249 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120255 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120260 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120265 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120269 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120273 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120278 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120282 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120286 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120290 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120294 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120298 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120304 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120308 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120312 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120317 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120321 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120325 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120329 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:26:04.133686 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120333 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120337 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120341 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120346 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120350 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120356 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120360 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120365 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120369 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120373 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120377 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120400 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120406 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120410 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120414 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120418 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120422 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120427 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120431 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:26:04.134152 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120435 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120439 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120443 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120447 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120451 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120455 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120459 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120464 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120469 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120473 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120477 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120481 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120486 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120490 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120494 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120498 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120502 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120507 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120513 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120517 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:26:04.134723 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120521 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120525 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120530 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120534 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120539 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120547 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120554 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120560 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120565 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120569 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120573 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120578 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120582 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120586 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120590 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120595 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120599 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120603 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120608 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:26:04.135234 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120612 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120617 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120622 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120626 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120630 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120634 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120638 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120643 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120647 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120651 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120655 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120661 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120695 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120703 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120708 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120712 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120716 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120722 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120727 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:26:04.135815 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120731 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:26:04.136322 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120735 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:26:04.136322 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120739 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:26:04.136322 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120743 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:26:04.136322 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120748 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:26:04.136322 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120752 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:26:04.136322 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120756 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:26:04.136322 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120760 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:26:04.136322 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:04.120765 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:26:04.136322 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.120838 2572 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 20:26:04.136322 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.124309 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 20:26:04.136322 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.125177 2572 server.go:1014] "Starting client certificate rotation" Oct 30 20:26:04.136322 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.125272 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:26:04.136322 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.125311 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:26:04.136322 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.127506 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:26:04.136322 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.131027 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:26:04.141679 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.141661 2572 log.go:25] "Validated CRI v1 runtime API" Oct 30 20:26:04.145124 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.145107 2572 log.go:25] "Validated CRI v1 image API" Oct 30 20:26:04.145966 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.145941 2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 20:26:04.149855 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.149829 2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c70833be-4520-4920-904c-a76540f204e5:/dev/nvme0n1p3 f5ce8070-923c-4036-9872-0a92e217993d:/dev/nvme0n1p4] Oct 30 20:26:04.149920 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.149855 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 20:26:04.154893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.154779 2572 manager.go:217] Machine: {Timestamp:2026-10-30 20:26:04.153523255 +0000 UTC m=+0.337738360 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097125 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22766d10a560c37c482443d2839b55 SystemUUID:ec22766d-10a5-60c3-7c48-2443d2839b55 BootID:acb9433a-3f30-4886-851b-4e9b11c2b1fe Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:66:2a:fa:a6:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:66:2a:fa:a6:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:3a:48:4f:4d:9d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 20:26:04.154893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.154891 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 20:26:04.155019 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.155007 2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 20:26:04.156071 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.156045 2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 20:26:04.156200 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.156074 2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-195.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 20:26:04.156248 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.156202 2572 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 20:26:04.156248 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.156211 2572 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 20:26:04.156248 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.156237 2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:26:04.156319 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.156253 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:26:04.157113 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.157103 2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 20:26:04.160760 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.160748 2572 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 20:26:04.160882 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.160873 2572 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 20:26:04.161583 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.161573 2572 kubelet.go:493] "Attempting to sync node with API server" Oct 30 20:26:04.161618 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.161588 2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 20:26:04.161618 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.161598 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 20:26:04.161618 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.161607 2572 kubelet.go:405] "Adding apiserver pod source" Oct 30 20:26:04.161618 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.161615 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 20:26:04.161942 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.161932 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:26:04.163961 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.163945 2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 20:26:04.169617 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.169605 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:26:04.170451 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.170439 2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 20:26:04.170486 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.170475 2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 20:26:04.170626 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.170617 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 20:26:04.170655 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.170633 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 20:26:04.170655 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.170640 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 20:26:04.170655 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.170646 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 20:26:04.170655 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.170651 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 20:26:04.170750 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.170657 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 20:26:04.171278 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.171269 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 20:26:04.171309 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.171278 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 20:26:04.171309 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.171286 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 20:26:04.171309 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.171291 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 20:26:04.171309 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.171299 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 20:26:04.171309 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.171308 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 20:26:04.171898 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.171890 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 20:26:04.171927 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.171899 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 20:26:04.175630 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.175616 2572 server.go:1263] "Started kubelet" Oct 30 20:26:04.175718 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.175689 2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 20:26:04.175802 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.175710 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 20:26:04.175896 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.175819 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 20:26:04.176284 ip-10-0-139-195 systemd[1]: Started Kubernetes Kubelet. Oct 30 20:26:04.177168 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.177155 2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 20:26:04.179535 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.179515 2572 server.go:317] "Adding debug handlers to kubelet server" Oct 30 20:26:04.181659 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.181633 2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 20:26:04.184530 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.184507 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 20:26:04.185117 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.185088 2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 20:26:04.185262 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.185243 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-195.us-west-1.compute.internal\" not found" Oct 30 20:26:04.185725 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.185708 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 20:26:04.185725 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.185708 2572 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 20:26:04.185832 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.185736 2572 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 20:26:04.185832 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.185822 2572 reconstruct.go:97] "Volume reconstruction finished" Oct 30 20:26:04.185832 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.185830 2572 reconciler.go:29] "Reconciler: start to sync state" Oct 30 20:26:04.185979 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.185910 2572 factory.go:55] Registering systemd factory Oct 30 20:26:04.185979 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.185927 2572 factory.go:223] Registration of the systemd container factory successfully Oct 30 20:26:04.186326 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.186303 2572 factory.go:153] Registering CRI-O factory Oct 30 20:26:04.186326 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.186321 2572 factory.go:223] Registration of the crio container factory successfully Oct 30 20:26:04.186425 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.186372 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 20:26:04.186463 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.186448 2572 factory.go:103] Registering Raw factory Oct 30 20:26:04.186491 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.186478 2572 manager.go:1199] Started watching for new ooms in manager Oct 30 20:26:04.187411 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.187379 2572 manager.go:319] Starting recovery of all containers Oct 30 20:26:04.196717 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.196701 2572 manager.go:324] Recovery completed Oct 30 20:26:04.200580 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.200568 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:26:04.202080 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.202063 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:26:04.202143 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.202096 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:26:04.202143 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.202111 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:26:04.202742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.202730 2572 cpu_manager.go:226] "Starting" policy="none" Oct 30 20:26:04.202742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.202737 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:26:04.202742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.202740 2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 20:26:04.202867 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.202757 2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 20:26:04.203471 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.203453 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:26:04.203519 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.203494 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:26:04.203519 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.203511 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:26:04.204991 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.204981 2572 policy_none.go:50] "Start" Oct 30 20:26:04.205024 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.204996 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 20:26:04.205024 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.205016 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 20:26:04.206978 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.206968 2572 policy_none.go:44] "Start" Oct 30 20:26:04.240029 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.239895 2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 20:26:04.244257 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.244244 2572 manager.go:341] "Starting Device Plugin manager" Oct 30 20:26:04.244342 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.244280 2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 20:26:04.244342 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.244292 2572 server.go:87] "Starting device plugin registration server" Oct 30 20:26:04.244520 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.244507 2572 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 20:26:04.244565 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.244519 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 20:26:04.244633 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.244621 2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 20:26:04.244705 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.244694 2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 20:26:04.244751 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.244706 2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 20:26:04.245200 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.245173 2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 20:26:04.245286 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.245223 2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-195.us-west-1.compute.internal\" not found" Oct 30 20:26:04.313025 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.312983 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 20:26:04.314750 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.314328 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 20:26:04.314840 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.314817 2572 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 20:26:04.314897 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.314864 2572 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 20:26:04.314944 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.314908 2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 20:26:04.326348 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.325300 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-195.us-west-1.compute.internal.18e3687d57b9e3e2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-195.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-195.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:26:04.17559037 +0000 UTC m=+0.359805470,LastTimestamp:2026-10-30 20:26:04.17559037 +0000 UTC m=+0.359805470,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-195.us-west-1.compute.internal,}" Oct 30 20:26:04.344768 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.344752 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:26:04.345535 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.345519 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:26:04.345596 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.345548 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:26:04.345596 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.345565 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:26:04.345596 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.345591 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.373425 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.373357 2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-195.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 20:26:04.373425 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.373414 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-195.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 20:26:04.378993 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.378917 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-195.us-west-1.compute.internal.18e3687d594e17b4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-195.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-195.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-195.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:26:04.20208018 +0000 UTC m=+0.386295279,LastTimestamp:2026-10-30 20:26:04.20208018 +0000 UTC m=+0.386295279,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-195.us-west-1.compute.internal,}" Oct 30 20:26:04.379323 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.379299 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-195.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 20:26:04.415655 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.415633 2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal"] Oct 30 20:26:04.415724 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.415685 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:26:04.416566 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.416550 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:26:04.416640 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.416579 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:26:04.416640 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.416590 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:26:04.418654 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.418642 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:26:04.418784 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.418771 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.418822 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.418798 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:26:04.419349 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.419325 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:26:04.419349 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.419342 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:26:04.419533 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.419358 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:26:04.419533 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.419365 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:26:04.419533 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.419372 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:26:04.419533 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.419379 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:26:04.421319 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.421304 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.421407 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.421332 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:26:04.421988 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.421975 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:26:04.422068 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.421997 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:26:04.422068 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.422024 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:26:04.450501 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.450479 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 20:26:04.450646 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.450544 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vwxpf" Oct 30 20:26:04.456274 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.456257 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-195.us-west-1.compute.internal\" not found" node="ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.458483 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.458466 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-195.us-west-1.compute.internal\" not found" node="ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.481086 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.481007 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-195.us-west-1.compute.internal.18e3687d594e70c9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-195.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-195.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-195.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:26:04.202102985 +0000 UTC m=+0.386318083,LastTimestamp:2026-10-30 20:26:04.202102985 +0000 UTC m=+0.386318083,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-195.us-west-1.compute.internal,}" Oct 30 20:26:04.481565 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.481543 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 20:26:04.481621 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.481542 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 20:26:04.488357 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.488338 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d006fe2d257556fbc453c5bdc6771005-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"d006fe2d257556fbc453c5bdc6771005\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.488454 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.488363 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d006fe2d257556fbc453c5bdc6771005-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"d006fe2d257556fbc453c5bdc6771005\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.488454 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.488379 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b1b5f93e01c03391b49a05418cff0a8e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"b1b5f93e01c03391b49a05418cff0a8e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.529822 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.529794 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 20:26:04.537881 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.537860 2572 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-139-195.us-west-1.compute.internal\" not found" node="ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.589138 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.589114 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d006fe2d257556fbc453c5bdc6771005-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"d006fe2d257556fbc453c5bdc6771005\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.589241 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.589138 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d006fe2d257556fbc453c5bdc6771005-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"d006fe2d257556fbc453c5bdc6771005\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.589241 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.589157 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b1b5f93e01c03391b49a05418cff0a8e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"b1b5f93e01c03391b49a05418cff0a8e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.589241 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.589183 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b1b5f93e01c03391b49a05418cff0a8e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"b1b5f93e01c03391b49a05418cff0a8e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.589241 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.589226 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d006fe2d257556fbc453c5bdc6771005-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"d006fe2d257556fbc453c5bdc6771005\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.589364 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.589226 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d006fe2d257556fbc453c5bdc6771005-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"d006fe2d257556fbc453c5bdc6771005\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.679807 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.679741 2572 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-195.us-west-1.compute.internal\" not found" node="ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.738690 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.738664 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:26:04.740346 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.740331 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:26:04.740437 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.740364 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:26:04.740437 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.740378 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:26:04.740437 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.740429 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.759539 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.759515 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.762555 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.762538 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.799088 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:04.799069 2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:04.799174 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.799090 2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-195.us-west-1.compute.internal\": node \"ip-10-0-139-195.us-west-1.compute.internal\" not found" Oct 30 20:26:04.928200 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:04.928173 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-195.us-west-1.compute.internal\" not found" Oct 30 20:26:05.028949 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:05.028868 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-195.us-west-1.compute.internal\" not found" Oct 30 20:26:05.125459 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:05.125433 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 20:26:05.125876 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:05.125561 2572 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a387d27a2dc53463097c2534c2d2347e-212d01575edb5520.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-195.us-west-1.compute.internal.18e3687d594e9e16\": read tcp 10.0.139.195:60428->18.225.152.9:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-195.us-west-1.compute.internal.18e3687d594e9e16 default 5208 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-195.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-195.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-195.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:26:04 +0000 UTC,LastTimestamp:2026-10-30 20:26:04.345570884 +0000 UTC m=+0.529785984,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-195.us-west-1.compute.internal,}" Oct 30 20:26:05.129727 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:05.129706 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-195.us-west-1.compute.internal\" not found" Oct 30 20:26:05.230399 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:05.230363 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-195.us-west-1.compute.internal\" not found" Oct 30 20:26:05.331240 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:05.331188 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-195.us-west-1.compute.internal\" not found" Oct 30 20:26:05.431842 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:05.431816 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-195.us-west-1.compute.internal\" not found" Oct 30 20:26:05.457816 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:05.457786 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:21:04 +0000 UTC" deadline="2028-07-29 18:42:04.523883524 +0000 UTC" Oct 30 20:26:05.457816 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:05.457812 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15310h15m59.066074184s" Oct 30 20:26:05.470963 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:05.470947 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:26:05.471798 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:05.471776 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:26:05.471886 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:05.471810 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:26:05.471886 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:05.471820 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:26:05.532202 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:05.532177 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-195.us-west-1.compute.internal\" not found" Oct 30 20:26:05.545771 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:05.545747 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:26:05.550202 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:05.550187 2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:26:05.585634 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:05.585579 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:05.648379 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:05.648357 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:26:05.649524 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:05.649511 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:26:05.708342 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:05.708322 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:26:05.741194 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:05.741173 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:26:05.799713 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:05.799682 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:26:05.799817 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:05.799711 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd006fe2d257556fbc453c5bdc6771005.slice/crio-4b9ee59e00f9f10151f45bc9eeda5290f733fa47f3e1c8e0aa5fde6fe999f48c WatchSource:0}: Error finding container 4b9ee59e00f9f10151f45bc9eeda5290f733fa47f3e1c8e0aa5fde6fe999f48c: Status 404 returned error can't find the container with id 4b9ee59e00f9f10151f45bc9eeda5290f733fa47f3e1c8e0aa5fde6fe999f48c Oct 30 20:26:05.805541 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:05.805527 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:26:05.825650 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:05.825622 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b5f93e01c03391b49a05418cff0a8e.slice/crio-81bd1aa7a4ea1b9b050060623a14ecd686bad9ce306fb444336a134c6c1c4b9d WatchSource:0}: Error finding container 81bd1aa7a4ea1b9b050060623a14ecd686bad9ce306fb444336a134c6c1c4b9d: Status 404 returned error can't find the container with id 81bd1aa7a4ea1b9b050060623a14ecd686bad9ce306fb444336a134c6c1c4b9d Oct 30 20:26:06.161870 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.161847 2572 apiserver.go:52] "Watching apiserver" Oct 30 20:26:06.186041 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.186015 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 20:26:06.255748 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.255675 2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 20:26:06.255925 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.255911 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-wwknf","openshift-image-registry/node-ca-zttcb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw","openshift-multus/multus-additional-cni-plugins-6txlq","openshift-multus/multus-c4qhb","kube-system/konnectivity-agent-6kfsl","openshift-multus/network-metrics-daemon-8sd9b","openshift-network-diagnostics/network-check-target-tw7jg","openshift-network-operator/iptables-alerter-9mq4r","kube-system/global-pull-secret-syncer-lwk8k","openshift-dns/node-resolver-vbtnk","openshift-ovn-kubernetes/ovnkube-node-6kqkm"] Oct 30 20:26:06.259050 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.259035 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:06.259102 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:06.259090 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:06.260881 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.260869 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:26:06.262735 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.262717 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:26:06.264722 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.264706 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.267623 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.267606 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.269654 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.269638 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.271646 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.271632 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:06.271712 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:06.271697 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:06.273644 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.273624 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.275661 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.275645 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.277752 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.277738 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:26:06.279830 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.279814 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:26:06.281656 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.281641 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:06.281723 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:06.281682 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:06.286905 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.286886 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 20:26:06.299306 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299286 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-ovnkube-config\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.299364 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299321 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-kubelet\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.299364 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299343 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-sys-fs\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.299439 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299416 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4fe6a899-6382-488a-b52d-e20c68445e67-hosts-file\") pod \"node-resolver-vbtnk\" (UID: \"4fe6a899-6382-488a-b52d-e20c68445e67\") " pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:26:06.299468 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299450 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe6a899-6382-488a-b52d-e20c68445e67-tmp-dir\") pod \"node-resolver-vbtnk\" (UID: \"4fe6a899-6382-488a-b52d-e20c68445e67\") " pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:26:06.299498 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299468 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wz4p\" (UniqueName: \"kubernetes.io/projected/4fe6a899-6382-488a-b52d-e20c68445e67-kube-api-access-9wz4p\") pod \"node-resolver-vbtnk\" (UID: \"4fe6a899-6382-488a-b52d-e20c68445e67\") " pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:26:06.299526 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299504 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-877qg\" (UniqueName: \"kubernetes.io/projected/d7a37dcb-96d5-4c31-a7d7-08f663036a86-kube-api-access-877qg\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:06.299554 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299541 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46932d2f-65ad-4ac4-b422-f4991d35cbd2-cni-binary-copy\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.299580 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299563 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysconfig\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.299615 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299597 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-lib-modules\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.299653 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299615 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-etc-selinux\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.299653 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299634 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-cni-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.299735 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299655 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-tuned\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.299735 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299671 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19997935-7a8e-4747-bb75-4532f1a72db5-ovn-node-metrics-cert\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.299735 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299714 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-ovnkube-script-lib\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.299816 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299746 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:06.299816 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299764 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.299816 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299780 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9whs\" (UniqueName: \"kubernetes.io/projected/590e650a-39b7-4696-8bf9-c105a87fb012-kube-api-access-w9whs\") pod \"iptables-alerter-9mq4r\" (UID: \"590e650a-39b7-4696-8bf9-c105a87fb012\") " pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:26:06.299816 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299792 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95f11ca1-5d71-4135-bf16-0fe6c24706c0-dbus\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:06.299911 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299824 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.299911 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299844 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-cnibin\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.299911 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299855 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-cni-bin\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.299911 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299877 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-kubelet\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.300012 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299950 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-var-lib-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.300012 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.299990 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.300067 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300031 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-env-overrides\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.300067 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300049 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-multus-certs\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.300118 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300068 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-tmp\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.300118 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300097 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4qm4\" (UniqueName: \"kubernetes.io/projected/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-kube-api-access-w4qm4\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.300185 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300114 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/590e650a-39b7-4696-8bf9-c105a87fb012-host-slash\") pod \"iptables-alerter-9mq4r\" (UID: \"590e650a-39b7-4696-8bf9-c105a87fb012\") " pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:26:06.300185 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300136 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-cni-netd\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.300185 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300170 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-registration-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.300321 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300191 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:06.300321 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300209 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-kubernetes\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.300321 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300252 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysctl-d\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.300321 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300283 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-sys\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.300321 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300305 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-run-netns\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.300542 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300328 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zkkn\" (UniqueName: \"kubernetes.io/projected/10982725-01b4-4d8e-8ad4-44d43608e6b0-kube-api-access-2zkkn\") pod \"node-ca-zttcb\" (UID: \"10982725-01b4-4d8e-8ad4-44d43608e6b0\") " pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:26:06.300542 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300353 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.300542 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300397 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-system-cni-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.300542 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300424 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-socket-dir-parent\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.300542 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300455 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-etc-kubernetes\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.300542 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300482 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4p2b\" (UniqueName: \"kubernetes.io/projected/46932d2f-65ad-4ac4-b422-f4991d35cbd2-kube-api-access-x4p2b\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.300542 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300523 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysctl-conf\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.300756 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300547 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgbs9\" (UniqueName: \"kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9\") pod \"network-check-target-tw7jg\" (UID: \"0001caf2-fffa-457e-8ab5-9574e11aced5\") " pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:06.300756 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300577 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-k8s-cni-cncf-io\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.300756 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300605 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-conf-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.300756 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300639 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-systemd-units\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.300756 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300669 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.300756 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300713 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-cni-bin\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.300756 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300741 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-var-lib-kubelet\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.300976 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300762 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.300976 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300797 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dff6e046-945d-4553-ba94-908dbb1a0c23-konnectivity-ca\") pod \"konnectivity-agent-6kfsl\" (UID: \"dff6e046-945d-4553-ba94-908dbb1a0c23\") " pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:26:06.300976 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300813 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10982725-01b4-4d8e-8ad4-44d43608e6b0-serviceca\") pod \"node-ca-zttcb\" (UID: \"10982725-01b4-4d8e-8ad4-44d43608e6b0\") " pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:26:06.300976 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300826 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-cni-multus\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.300976 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300839 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-slash\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.300976 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300880 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-etc-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.300976 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300922 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-log-socket\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.300976 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300951 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.301217 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.300981 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-socket-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.301217 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301004 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-system-cni-dir\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.301217 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301040 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-host\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.301217 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301065 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-device-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.301217 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301100 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95f11ca1-5d71-4135-bf16-0fe6c24706c0-kubelet-config\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:06.301217 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301125 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-netns\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.301217 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301163 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-daemon-config\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.301217 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301188 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-modprobe-d\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.301217 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301203 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-node-log\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.301571 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301225 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dff6e046-945d-4553-ba94-908dbb1a0c23-agent-certs\") pod \"konnectivity-agent-6kfsl\" (UID: \"dff6e046-945d-4553-ba94-908dbb1a0c23\") " pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:26:06.301571 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301260 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-run\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.301571 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301291 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h2ng\" (UniqueName: \"kubernetes.io/projected/0ddcb000-6271-4c42-a911-b3a4d14711c3-kube-api-access-5h2ng\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.301571 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301314 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-cnibin\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.301571 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301339 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-hostroot\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.301571 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301363 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-systemd\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.301571 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301419 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/590e650a-39b7-4696-8bf9-c105a87fb012-iptables-alerter-script\") pod \"iptables-alerter-9mq4r\" (UID: \"590e650a-39b7-4696-8bf9-c105a87fb012\") " pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:26:06.301571 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301449 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdrp4\" (UniqueName: \"kubernetes.io/projected/6a305968-1cca-480f-9360-aeb99c0ef2de-kube-api-access-pdrp4\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.301571 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301471 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.301571 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301496 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g72wk\" (UniqueName: \"kubernetes.io/projected/19997935-7a8e-4747-bb75-4532f1a72db5-kube-api-access-g72wk\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.301571 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301510 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10982725-01b4-4d8e-8ad4-44d43608e6b0-host\") pod \"node-ca-zttcb\" (UID: \"10982725-01b4-4d8e-8ad4-44d43608e6b0\") " pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:26:06.301571 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301528 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-systemd\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.301571 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.301551 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-ovn\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.311006 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.310985 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 20:26:06.311485 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.311462 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 20:26:06.311581 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.311565 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 20:26:06.311640 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.311580 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wczr5\"" Oct 30 20:26:06.311640 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.311627 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 20:26:06.311726 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.311657 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9wdmm\"" Oct 30 20:26:06.312207 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.312188 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 20:26:06.314979 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.314950 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 20:26:06.315058 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.314978 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 20:26:06.315058 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.315027 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 20:26:06.315058 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.315051 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-47trg\"" Oct 30 20:26:06.315188 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.315139 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 20:26:06.315352 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.315334 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 20:26:06.315352 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.315296 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bx2zg\"" Oct 30 20:26:06.315504 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.315408 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 20:26:06.315537 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.315514 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 20:26:06.315673 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.315662 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 20:26:06.316689 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.316675 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 20:26:06.318093 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.318077 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ld6lr\"" Oct 30 20:26:06.318452 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.318437 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 20:26:06.318522 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.318499 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 20:26:06.318522 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.318516 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 20:26:06.318607 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.318556 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 20:26:06.318680 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.318650 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" event={"ID":"b1b5f93e01c03391b49a05418cff0a8e","Type":"ContainerStarted","Data":"81bd1aa7a4ea1b9b050060623a14ecd686bad9ce306fb444336a134c6c1c4b9d"} Oct 30 20:26:06.319555 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.319538 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" event={"ID":"d006fe2d257556fbc453c5bdc6771005","Type":"ContainerStarted","Data":"4b9ee59e00f9f10151f45bc9eeda5290f733fa47f3e1c8e0aa5fde6fe999f48c"} Oct 30 20:26:06.353085 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.353066 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 20:26:06.353160 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.353107 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 20:26:06.356520 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.356505 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 20:26:06.356563 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.356553 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kmj5c" Oct 30 20:26:06.402143 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402119 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-ovn\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.402222 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402146 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-ovnkube-config\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.402222 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402162 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-kubelet\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.402222 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402182 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-sys-fs\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.402222 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402203 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4fe6a899-6382-488a-b52d-e20c68445e67-hosts-file\") pod \"node-resolver-vbtnk\" (UID: \"4fe6a899-6382-488a-b52d-e20c68445e67\") " pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:26:06.402457 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402223 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe6a899-6382-488a-b52d-e20c68445e67-tmp-dir\") pod \"node-resolver-vbtnk\" (UID: \"4fe6a899-6382-488a-b52d-e20c68445e67\") " pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:26:06.402457 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402243 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wz4p\" (UniqueName: \"kubernetes.io/projected/4fe6a899-6382-488a-b52d-e20c68445e67-kube-api-access-9wz4p\") pod \"node-resolver-vbtnk\" (UID: \"4fe6a899-6382-488a-b52d-e20c68445e67\") " pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:26:06.402457 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402250 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-kubelet\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.402457 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402265 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-877qg\" (UniqueName: \"kubernetes.io/projected/d7a37dcb-96d5-4c31-a7d7-08f663036a86-kube-api-access-877qg\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:06.402457 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402252 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-ovn\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.402457 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402295 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4fe6a899-6382-488a-b52d-e20c68445e67-hosts-file\") pod \"node-resolver-vbtnk\" (UID: \"4fe6a899-6382-488a-b52d-e20c68445e67\") " pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:26:06.402457 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402305 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-sys-fs\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.402457 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402406 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46932d2f-65ad-4ac4-b422-f4991d35cbd2-cni-binary-copy\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.402457 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402428 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 20:26:06.402457 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402433 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysconfig\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.402742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402481 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-lib-modules\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.402742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402503 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-etc-selinux\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.402742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402532 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qmclr\"" Oct 30 20:26:06.402742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402548 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-cni-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.402742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402572 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysconfig\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.402742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402569 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-tuned\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.402742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402606 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 20:26:06.402742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402625 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19997935-7a8e-4747-bb75-4532f1a72db5-ovn-node-metrics-cert\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.402742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402651 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-ovnkube-script-lib\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.402742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402669 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:06.402742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402697 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.402742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402704 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-etc-selinux\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.402742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402723 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9whs\" (UniqueName: \"kubernetes.io/projected/590e650a-39b7-4696-8bf9-c105a87fb012-kube-api-access-w9whs\") pod \"iptables-alerter-9mq4r\" (UID: \"590e650a-39b7-4696-8bf9-c105a87fb012\") " pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402747 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-cni-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402743 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95f11ca1-5d71-4135-bf16-0fe6c24706c0-dbus\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402759 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402780 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402800 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-lib-modules\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402801 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-cnibin\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402820 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-52gtb\"" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402837 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402837 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-cni-bin\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402865 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-kubelet\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402886 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-var-lib-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402892 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402909 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402931 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-env-overrides\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402944 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402953 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-multus-certs\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.402974 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-tmp\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403002 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4qm4\" (UniqueName: \"kubernetes.io/projected/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-kube-api-access-w4qm4\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403019 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fngvs\"" Oct 30 20:26:06.403051 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403032 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/590e650a-39b7-4696-8bf9-c105a87fb012-host-slash\") pod \"iptables-alerter-9mq4r\" (UID: \"590e650a-39b7-4696-8bf9-c105a87fb012\") " pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403062 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403048 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-cni-netd\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403087 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-cni-netd\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403096 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-registration-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403121 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403145 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-cnibin\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403144 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-kubernetes\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403171 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysctl-d\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403188 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-kubernetes\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403193 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-sys\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403213 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe6a899-6382-488a-b52d-e20c68445e67-tmp-dir\") pod \"node-resolver-vbtnk\" (UID: \"4fe6a899-6382-488a-b52d-e20c68445e67\") " pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403236 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-registration-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403122 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cnj9l\"" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403257 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-cni-bin\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403286 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-kubelet\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:06.403322 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403331 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95f11ca1-5d71-4135-bf16-0fe6c24706c0-dbus\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403334 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-var-lib-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.403850 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403364 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:06.403401 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs podName:d7a37dcb-96d5-4c31-a7d7-08f663036a86 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:06.903357965 +0000 UTC m=+3.087573079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs") pod "network-metrics-daemon-8sd9b" (UID: "d7a37dcb-96d5-4c31-a7d7-08f663036a86") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403214 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-run-netns\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403435 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zkkn\" (UniqueName: \"kubernetes.io/projected/10982725-01b4-4d8e-8ad4-44d43608e6b0-kube-api-access-2zkkn\") pod \"node-ca-zttcb\" (UID: \"10982725-01b4-4d8e-8ad4-44d43608e6b0\") " pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403461 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403461 2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403487 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-system-cni-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403509 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-socket-dir-parent\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403532 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-etc-kubernetes\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403554 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4p2b\" (UniqueName: \"kubernetes.io/projected/46932d2f-65ad-4ac4-b422-f4991d35cbd2-kube-api-access-x4p2b\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403571 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-run-netns\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403577 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysctl-conf\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403580 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46932d2f-65ad-4ac4-b422-f4991d35cbd2-cni-binary-copy\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403592 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-ovnkube-config\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403608 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbs9\" (UniqueName: \"kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9\") pod \"network-check-target-tw7jg\" (UID: \"0001caf2-fffa-457e-8ab5-9574e11aced5\") " pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403633 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-k8s-cni-cncf-io\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403649 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-system-cni-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.404809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403685 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysctl-conf\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403656 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-conf-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:06.403691 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403718 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-systemd-units\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403725 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-env-overrides\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:06.403747 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret podName:95f11ca1-5d71-4135-bf16-0fe6c24706c0 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:06.903732454 +0000 UTC m=+3.087947564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret") pod "global-pull-secret-syncer-lwk8k" (UID: "95f11ca1-5d71-4135-bf16-0fe6c24706c0") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403764 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403768 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-systemd-units\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403770 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-multus-certs\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403800 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-cni-bin\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403805 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-etc-kubernetes\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403852 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-socket-dir-parent\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403886 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-cni-bin\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403891 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/590e650a-39b7-4696-8bf9-c105a87fb012-host-slash\") pod \"iptables-alerter-9mq4r\" (UID: \"590e650a-39b7-4696-8bf9-c105a87fb012\") " pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403906 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-var-lib-kubelet\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403926 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403936 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-sys\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.405558 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403945 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dff6e046-945d-4553-ba94-908dbb1a0c23-konnectivity-ca\") pod \"konnectivity-agent-6kfsl\" (UID: \"dff6e046-945d-4553-ba94-908dbb1a0c23\") " pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403967 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10982725-01b4-4d8e-8ad4-44d43608e6b0-serviceca\") pod \"node-ca-zttcb\" (UID: \"10982725-01b4-4d8e-8ad4-44d43608e6b0\") " pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403978 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysctl-d\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403985 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-cni-multus\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404004 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-k8s-cni-cncf-io\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404012 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-slash\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404025 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-cni-multus\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404042 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-slash\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404051 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-etc-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.403853 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404082 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-log-socket\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404106 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404128 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-socket-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404133 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404164 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-system-cni-dir\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404188 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-ovnkube-script-lib\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404236 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-socket-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.406284 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404247 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-log-socket\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404254 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-host\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404283 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-system-cni-dir\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404288 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-etc-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404320 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-host\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404445 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dff6e046-945d-4553-ba94-908dbb1a0c23-konnectivity-ca\") pod \"konnectivity-agent-6kfsl\" (UID: \"dff6e046-945d-4553-ba94-908dbb1a0c23\") " pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404456 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404449 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10982725-01b4-4d8e-8ad4-44d43608e6b0-serviceca\") pod \"node-ca-zttcb\" (UID: \"10982725-01b4-4d8e-8ad4-44d43608e6b0\") " pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404487 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-device-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404489 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-var-lib-kubelet\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404510 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95f11ca1-5d71-4135-bf16-0fe6c24706c0-kubelet-config\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404529 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-netns\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404547 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-daemon-config\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404563 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-device-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404564 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-modprobe-d\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404601 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-node-log\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404621 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-modprobe-d\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404622 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dff6e046-945d-4553-ba94-908dbb1a0c23-agent-certs\") pod \"konnectivity-agent-6kfsl\" (UID: \"dff6e046-945d-4553-ba94-908dbb1a0c23\") " pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:26:06.406893 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404649 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-run\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404678 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2ng\" (UniqueName: \"kubernetes.io/projected/0ddcb000-6271-4c42-a911-b3a4d14711c3-kube-api-access-5h2ng\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404697 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-cnibin\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404721 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-hostroot\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404744 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-systemd\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404796 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/590e650a-39b7-4696-8bf9-c105a87fb012-iptables-alerter-script\") pod \"iptables-alerter-9mq4r\" (UID: \"590e650a-39b7-4696-8bf9-c105a87fb012\") " pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404821 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdrp4\" (UniqueName: \"kubernetes.io/projected/6a305968-1cca-480f-9360-aeb99c0ef2de-kube-api-access-pdrp4\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404843 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404866 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g72wk\" (UniqueName: \"kubernetes.io/projected/19997935-7a8e-4747-bb75-4532f1a72db5-kube-api-access-g72wk\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404888 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10982725-01b4-4d8e-8ad4-44d43608e6b0-host\") pod \"node-ca-zttcb\" (UID: \"10982725-01b4-4d8e-8ad4-44d43608e6b0\") " pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404908 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-systemd\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404949 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404972 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-systemd\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.404978 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.405002 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-conf-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.405015 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95f11ca1-5d71-4135-bf16-0fe6c24706c0-kubelet-config\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.405021 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-node-log\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.407338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.405048 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-hostroot\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.407811 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.405067 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-systemd\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.407811 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.405084 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-netns\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.407811 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.405236 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.407811 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.405477 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10982725-01b4-4d8e-8ad4-44d43608e6b0-host\") pod \"node-ca-zttcb\" (UID: \"10982725-01b4-4d8e-8ad4-44d43608e6b0\") " pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:26:06.407811 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.405523 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-daemon-config\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.407811 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.405551 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/590e650a-39b7-4696-8bf9-c105a87fb012-iptables-alerter-script\") pod \"iptables-alerter-9mq4r\" (UID: \"590e650a-39b7-4696-8bf9-c105a87fb012\") " pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:26:06.407811 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.405578 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-cnibin\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.407811 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.405605 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-run\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.407811 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.406318 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.407811 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.406756 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-tuned\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.407811 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.406880 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-tmp\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.407811 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.406955 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19997935-7a8e-4747-bb75-4532f1a72db5-ovn-node-metrics-cert\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.407811 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.407058 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dff6e046-945d-4553-ba94-908dbb1a0c23-agent-certs\") pod \"konnectivity-agent-6kfsl\" (UID: \"dff6e046-945d-4553-ba94-908dbb1a0c23\") " pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:26:06.456110 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.456091 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wz4p\" (UniqueName: \"kubernetes.io/projected/4fe6a899-6382-488a-b52d-e20c68445e67-kube-api-access-9wz4p\") pod \"node-resolver-vbtnk\" (UID: \"4fe6a899-6382-488a-b52d-e20c68445e67\") " pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:26:06.457172 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:06.457148 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:26:06.457308 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:06.457177 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:26:06.457308 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:06.457191 2572 projected.go:196] Error preparing data for projected volume kube-api-access-wgbs9 for pod openshift-network-diagnostics/network-check-target-tw7jg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:26:06.457308 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:06.457261 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9 podName:0001caf2-fffa-457e-8ab5-9574e11aced5 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:06.957243764 +0000 UTC m=+3.141458863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wgbs9" (UniqueName: "kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9") pod "network-check-target-tw7jg" (UID: "0001caf2-fffa-457e-8ab5-9574e11aced5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:26:06.457916 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.457897 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9whs\" (UniqueName: \"kubernetes.io/projected/590e650a-39b7-4696-8bf9-c105a87fb012-kube-api-access-w9whs\") pod \"iptables-alerter-9mq4r\" (UID: \"590e650a-39b7-4696-8bf9-c105a87fb012\") " pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:26:06.457992 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.457895 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-877qg\" (UniqueName: \"kubernetes.io/projected/d7a37dcb-96d5-4c31-a7d7-08f663036a86-kube-api-access-877qg\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:06.458031 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.458003 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zkkn\" (UniqueName: \"kubernetes.io/projected/10982725-01b4-4d8e-8ad4-44d43608e6b0-kube-api-access-2zkkn\") pod \"node-ca-zttcb\" (UID: \"10982725-01b4-4d8e-8ad4-44d43608e6b0\") " pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:26:06.458482 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.458469 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4qm4\" (UniqueName: \"kubernetes.io/projected/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-kube-api-access-w4qm4\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.459257 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.459243 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4p2b\" (UniqueName: \"kubernetes.io/projected/46932d2f-65ad-4ac4-b422-f4991d35cbd2-kube-api-access-x4p2b\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.461951 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.461933 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h2ng\" (UniqueName: \"kubernetes.io/projected/0ddcb000-6271-4c42-a911-b3a4d14711c3-kube-api-access-5h2ng\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.462142 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.462125 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g72wk\" (UniqueName: \"kubernetes.io/projected/19997935-7a8e-4747-bb75-4532f1a72db5-kube-api-access-g72wk\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.462178 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.462138 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdrp4\" (UniqueName: \"kubernetes.io/projected/6a305968-1cca-480f-9360-aeb99c0ef2de-kube-api-access-pdrp4\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.571033 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.570954 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:26:06.576892 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.576877 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:26:06.580842 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:06.580819 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe6a899_6382_488a_b52d_e20c68445e67.slice/crio-5fd458ac66e32419632ae0455910180d935049c4b6fdbd6134b35228130793f2 WatchSource:0}: Error finding container 5fd458ac66e32419632ae0455910180d935049c4b6fdbd6134b35228130793f2: Status 404 returned error can't find the container with id 5fd458ac66e32419632ae0455910180d935049c4b6fdbd6134b35228130793f2 Oct 30 20:26:06.584816 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:06.584796 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10982725_01b4_4d8e_8ad4_44d43608e6b0.slice/crio-4dbfa8cf88c5a111ad23499c0d035296b7b8e9a30edd98d6e13536a6b620899b WatchSource:0}: Error finding container 4dbfa8cf88c5a111ad23499c0d035296b7b8e9a30edd98d6e13536a6b620899b: Status 404 returned error can't find the container with id 4dbfa8cf88c5a111ad23499c0d035296b7b8e9a30edd98d6e13536a6b620899b Oct 30 20:26:06.598938 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.598923 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:26:06.601985 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.601970 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c4qhb" Oct 30 20:26:06.606808 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.605269 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:06.607434 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.607419 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:26:06.609767 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:06.609749 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a305968_1cca_480f_9360_aeb99c0ef2de.slice/crio-365cb554216b8fa9574c9b4c326fc8927b9d610d05f3477e9fa04a4462aaf968 WatchSource:0}: Error finding container 365cb554216b8fa9574c9b4c326fc8927b9d610d05f3477e9fa04a4462aaf968: Status 404 returned error can't find the container with id 365cb554216b8fa9574c9b4c326fc8927b9d610d05f3477e9fa04a4462aaf968 Oct 30 20:26:06.612764 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:06.612745 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46932d2f_65ad_4ac4_b422_f4991d35cbd2.slice/crio-947e91e49a01c04511694a679764c49788f373f6bf7e637231fd11bc0f5f862b WatchSource:0}: Error finding container 947e91e49a01c04511694a679764c49788f373f6bf7e637231fd11bc0f5f862b: Status 404 returned error can't find the container with id 947e91e49a01c04511694a679764c49788f373f6bf7e637231fd11bc0f5f862b Oct 30 20:26:06.613671 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.613658 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:26:06.615944 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.615848 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:26:06.616065 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:06.616047 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19997935_7a8e_4747_bb75_4532f1a72db5.slice/crio-13f03e50512a578a5ec3e178a77dcb47dc4637aa5fe2f0fe074b2e67172dba83 WatchSource:0}: Error finding container 13f03e50512a578a5ec3e178a77dcb47dc4637aa5fe2f0fe074b2e67172dba83: Status 404 returned error can't find the container with id 13f03e50512a578a5ec3e178a77dcb47dc4637aa5fe2f0fe074b2e67172dba83 Oct 30 20:26:06.618208 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:06.618166 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ac6e99_4947_4703_a31d_091a6cc9b9d7.slice/crio-aa07ce351ee07e4b4cc482e6e89d104da9b8e2bf5671ed10b28ab4df75553bb2 WatchSource:0}: Error finding container aa07ce351ee07e4b4cc482e6e89d104da9b8e2bf5671ed10b28ab4df75553bb2: Status 404 returned error can't find the container with id aa07ce351ee07e4b4cc482e6e89d104da9b8e2bf5671ed10b28ab4df75553bb2 Oct 30 20:26:06.621645 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.621627 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:26:06.622845 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:06.622827 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff6e046_945d_4553_ba94_908dbb1a0c23.slice/crio-e78b32209ac052ac75dd366a4bfdbe197ee2de2c77c81bbc38acd54b4b9b3a75 WatchSource:0}: Error finding container e78b32209ac052ac75dd366a4bfdbe197ee2de2c77c81bbc38acd54b4b9b3a75: Status 404 returned error can't find the container with id e78b32209ac052ac75dd366a4bfdbe197ee2de2c77c81bbc38acd54b4b9b3a75 Oct 30 20:26:06.624700 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:06.624681 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590e650a_39b7_4696_8bf9_c105a87fb012.slice/crio-e1b511941563444d88c34017d11fca27ce03fbfd82d708b2aff9eeb6c6c2f534 WatchSource:0}: Error finding container e1b511941563444d88c34017d11fca27ce03fbfd82d708b2aff9eeb6c6c2f534: Status 404 returned error can't find the container with id e1b511941563444d88c34017d11fca27ce03fbfd82d708b2aff9eeb6c6c2f534 Oct 30 20:26:06.630909 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:06.630891 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddcb000_6271_4c42_a911_b3a4d14711c3.slice/crio-879c830c59794efe325ec0e1c7671029cb1e7e5fdfd7af2736ab352f6eab8cba WatchSource:0}: Error finding container 879c830c59794efe325ec0e1c7671029cb1e7e5fdfd7af2736ab352f6eab8cba: Status 404 returned error can't find the container with id 879c830c59794efe325ec0e1c7671029cb1e7e5fdfd7af2736ab352f6eab8cba Oct 30 20:26:06.907926 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.907849 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:06.907926 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:06.907893 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:06.908170 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:06.907992 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:26:06.908170 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:06.908043 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret podName:95f11ca1-5d71-4135-bf16-0fe6c24706c0 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:07.908032306 +0000 UTC m=+4.092247391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret") pod "global-pull-secret-syncer-lwk8k" (UID: "95f11ca1-5d71-4135-bf16-0fe6c24706c0") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:26:06.908170 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:06.907992 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:26:06.908170 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:06.908084 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs podName:d7a37dcb-96d5-4c31-a7d7-08f663036a86 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:07.908075034 +0000 UTC m=+4.092290122 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs") pod "network-metrics-daemon-8sd9b" (UID: "d7a37dcb-96d5-4c31-a7d7-08f663036a86") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:26:07.008286 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:07.008255 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbs9\" (UniqueName: \"kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9\") pod \"network-check-target-tw7jg\" (UID: \"0001caf2-fffa-457e-8ab5-9574e11aced5\") " pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:07.008468 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:07.008452 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:26:07.008516 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:07.008490 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:26:07.008516 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:07.008501 2572 projected.go:196] Error preparing data for projected volume kube-api-access-wgbs9 for pod openshift-network-diagnostics/network-check-target-tw7jg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:26:07.008576 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:07.008563 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9 podName:0001caf2-fffa-457e-8ab5-9574e11aced5 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:08.008545913 +0000 UTC m=+4.192761004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wgbs9" (UniqueName: "kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9") pod "network-check-target-tw7jg" (UID: "0001caf2-fffa-457e-8ab5-9574e11aced5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:26:07.322580 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:07.322513 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zttcb" event={"ID":"10982725-01b4-4d8e-8ad4-44d43608e6b0","Type":"ContainerStarted","Data":"4dbfa8cf88c5a111ad23499c0d035296b7b8e9a30edd98d6e13536a6b620899b"} Oct 30 20:26:07.324035 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:07.323819 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vbtnk" event={"ID":"4fe6a899-6382-488a-b52d-e20c68445e67","Type":"ContainerStarted","Data":"5fd458ac66e32419632ae0455910180d935049c4b6fdbd6134b35228130793f2"} Oct 30 20:26:07.325071 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:07.325043 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txlq" event={"ID":"0ddcb000-6271-4c42-a911-b3a4d14711c3","Type":"ContainerStarted","Data":"879c830c59794efe325ec0e1c7671029cb1e7e5fdfd7af2736ab352f6eab8cba"} Oct 30 20:26:07.326349 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:07.326304 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6kfsl" event={"ID":"dff6e046-945d-4553-ba94-908dbb1a0c23","Type":"ContainerStarted","Data":"e78b32209ac052ac75dd366a4bfdbe197ee2de2c77c81bbc38acd54b4b9b3a75"} Oct 30 20:26:07.327526 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:07.327469 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wwknf" event={"ID":"e0ac6e99-4947-4703-a31d-091a6cc9b9d7","Type":"ContainerStarted","Data":"aa07ce351ee07e4b4cc482e6e89d104da9b8e2bf5671ed10b28ab4df75553bb2"} Oct 30 20:26:07.328742 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:07.328723 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c4qhb" event={"ID":"46932d2f-65ad-4ac4-b422-f4991d35cbd2","Type":"ContainerStarted","Data":"947e91e49a01c04511694a679764c49788f373f6bf7e637231fd11bc0f5f862b"} Oct 30 20:26:07.330086 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:07.330057 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" event={"ID":"6a305968-1cca-480f-9360-aeb99c0ef2de","Type":"ContainerStarted","Data":"365cb554216b8fa9574c9b4c326fc8927b9d610d05f3477e9fa04a4462aaf968"} Oct 30 20:26:07.331222 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:07.331200 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9mq4r" event={"ID":"590e650a-39b7-4696-8bf9-c105a87fb012","Type":"ContainerStarted","Data":"e1b511941563444d88c34017d11fca27ce03fbfd82d708b2aff9eeb6c6c2f534"} Oct 30 20:26:07.332427 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:07.332379 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" event={"ID":"19997935-7a8e-4747-bb75-4532f1a72db5","Type":"ContainerStarted","Data":"13f03e50512a578a5ec3e178a77dcb47dc4637aa5fe2f0fe074b2e67172dba83"} Oct 30 20:26:07.359783 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:07.359729 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:21:06 +0000 UTC" deadline="2028-03-27 11:26:52.571512868 +0000 UTC" Oct 30 20:26:07.359783 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:07.359780 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12327h0m45.211736646s" Oct 30 20:26:07.916151 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:07.916120 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:07.916325 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:07.916172 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:07.916325 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:07.916312 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:26:07.916477 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:07.916374 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs podName:d7a37dcb-96d5-4c31-a7d7-08f663036a86 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:09.916358095 +0000 UTC m=+6.100573183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs") pod "network-metrics-daemon-8sd9b" (UID: "d7a37dcb-96d5-4c31-a7d7-08f663036a86") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:26:07.916477 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:07.916457 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:26:07.916576 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:07.916500 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret podName:95f11ca1-5d71-4135-bf16-0fe6c24706c0 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:09.916480221 +0000 UTC m=+6.100695308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret") pod "global-pull-secret-syncer-lwk8k" (UID: "95f11ca1-5d71-4135-bf16-0fe6c24706c0") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:26:08.016909 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:08.016875 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbs9\" (UniqueName: \"kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9\") pod \"network-check-target-tw7jg\" (UID: \"0001caf2-fffa-457e-8ab5-9574e11aced5\") " pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:08.017089 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:08.017061 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:26:08.017089 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:08.017079 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:26:08.017089 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:08.017090 2572 projected.go:196] Error preparing data for projected volume kube-api-access-wgbs9 for pod openshift-network-diagnostics/network-check-target-tw7jg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:26:08.017252 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:08.017142 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9 podName:0001caf2-fffa-457e-8ab5-9574e11aced5 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:10.017125005 +0000 UTC m=+6.201340106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wgbs9" (UniqueName: "kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9") pod "network-check-target-tw7jg" (UID: "0001caf2-fffa-457e-8ab5-9574e11aced5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:26:08.318199 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:08.318126 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:08.318199 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:08.318167 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:08.318478 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:08.318251 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:08.318478 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:08.318133 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:08.318478 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:08.318351 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:08.318478 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:08.318442 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:08.360515 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:08.360452 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:21:06 +0000 UTC" deadline="2028-06-23 20:40:18.69268475 +0000 UTC" Oct 30 20:26:08.360515 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:08.360487 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14448h14m10.332203412s" Oct 30 20:26:09.350709 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:09.350627 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" event={"ID":"b1b5f93e01c03391b49a05418cff0a8e","Type":"ContainerStarted","Data":"2961efb98d549f8298cc3d7048c811fcb8ee8cb4b68a1c9f4a848a899d5261c6"} Oct 30 20:26:09.360084 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:09.359614 2572 generic.go:356] "Generic (PLEG): container finished" podID="d006fe2d257556fbc453c5bdc6771005" containerID="656fca1d33c016d430c96965802b5e13ad7eb2c48d2c6a7acde0d2c380e4572f" exitCode=0 Oct 30 20:26:09.360084 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:09.359743 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" event={"ID":"d006fe2d257556fbc453c5bdc6771005","Type":"ContainerDied","Data":"656fca1d33c016d430c96965802b5e13ad7eb2c48d2c6a7acde0d2c380e4572f"} Oct 30 20:26:09.576970 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:09.576896 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" podStartSLOduration=4.576876877 podStartE2EDuration="4.576876877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:09.463352184 +0000 UTC m=+5.647567293" watchObservedRunningTime="2026-10-30 20:26:09.576876877 +0000 UTC m=+5.761091987" Oct 30 20:26:09.934219 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:09.933815 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:09.934219 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:09.933866 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:09.934219 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:09.934017 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:26:09.934219 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:09.934079 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs podName:d7a37dcb-96d5-4c31-a7d7-08f663036a86 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:13.934059895 +0000 UTC m=+10.118275003 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs") pod "network-metrics-daemon-8sd9b" (UID: "d7a37dcb-96d5-4c31-a7d7-08f663036a86") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:26:09.934219 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:09.934151 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:26:09.934219 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:09.934187 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret podName:95f11ca1-5d71-4135-bf16-0fe6c24706c0 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:13.934175394 +0000 UTC m=+10.118390495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret") pod "global-pull-secret-syncer-lwk8k" (UID: "95f11ca1-5d71-4135-bf16-0fe6c24706c0") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:26:10.034704 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:10.034672 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbs9\" (UniqueName: \"kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9\") pod \"network-check-target-tw7jg\" (UID: \"0001caf2-fffa-457e-8ab5-9574e11aced5\") " pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:10.034860 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:10.034843 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:26:10.034936 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:10.034866 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:26:10.034936 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:10.034878 2572 projected.go:196] Error preparing data for projected volume kube-api-access-wgbs9 for pod openshift-network-diagnostics/network-check-target-tw7jg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:26:10.034936 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:10.034933 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9 podName:0001caf2-fffa-457e-8ab5-9574e11aced5 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:14.034916303 +0000 UTC m=+10.219131394 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wgbs9" (UniqueName: "kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9") pod "network-check-target-tw7jg" (UID: "0001caf2-fffa-457e-8ab5-9574e11aced5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:26:10.320622 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:10.317959 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:10.320622 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:10.318100 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:10.320622 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:10.318497 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:10.320622 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:10.318594 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:10.320622 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:10.318616 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:10.320622 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:10.318661 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:10.370579 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:10.370538 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" event={"ID":"d006fe2d257556fbc453c5bdc6771005","Type":"ContainerStarted","Data":"3e46fd0c758fa69a4841bcf5a5d96e5058eab27a1ccea0b2052c03e0cf7fa44c"} Oct 30 20:26:10.376933 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:10.376899 2572 generic.go:356] "Generic (PLEG): container finished" podID="0ddcb000-6271-4c42-a911-b3a4d14711c3" containerID="c2598612c921be9a11a3e45156726f45c2bd1ccdbb6e3acdeb5fa454265b7c5f" exitCode=0 Oct 30 20:26:10.377911 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:10.377884 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txlq" event={"ID":"0ddcb000-6271-4c42-a911-b3a4d14711c3","Type":"ContainerDied","Data":"c2598612c921be9a11a3e45156726f45c2bd1ccdbb6e3acdeb5fa454265b7c5f"} Oct 30 20:26:10.605711 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:10.605605 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" podStartSLOduration=5.605589484 podStartE2EDuration="5.605589484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:10.485095958 +0000 UTC m=+6.669311095" watchObservedRunningTime="2026-10-30 20:26:10.605589484 +0000 UTC m=+6.789804592" Oct 30 20:26:11.383014 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:11.382936 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6kfsl" event={"ID":"dff6e046-945d-4553-ba94-908dbb1a0c23","Type":"ContainerStarted","Data":"5e3610342385f0d5752842a88ee97b14c20deef6bb84bcac6a39c72e20fb933a"} Oct 30 20:26:12.318090 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:12.318012 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:12.318555 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:12.318160 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:12.318615 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:12.318580 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:12.318716 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:12.318685 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:12.318771 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:12.318750 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:12.318821 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:12.318796 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:12.388671 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:12.388628 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zttcb" event={"ID":"10982725-01b4-4d8e-8ad4-44d43608e6b0","Type":"ContainerStarted","Data":"5232d650fd00398cfddd8dc75297f6804d1cfd0488df2c7473537754110f0188"} Oct 30 20:26:12.503587 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:12.503501 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6kfsl" podStartSLOduration=4.023341383 podStartE2EDuration="8.503479853s" totalImagesPullingTime="4.48013847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:11.503335119 +0000 UTC m=+7.687550217" watchObservedRunningTime="2026-10-30 20:26:12.503479853 +0000 UTC m=+8.687694965" Oct 30 20:26:12.504188 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:12.504145 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zttcb" podStartSLOduration=3.16052617 podStartE2EDuration="8.504134933s" totalImagesPullingTime="5.343608763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:12.502821857 +0000 UTC m=+8.687036967" watchObservedRunningTime="2026-10-30 20:26:12.504134933 +0000 UTC m=+8.688350046" Oct 30 20:26:13.392829 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:13.392749 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" event={"ID":"6a305968-1cca-480f-9360-aeb99c0ef2de","Type":"ContainerStarted","Data":"ba89b8fd5eeac283843d961f20b4406143f778998ea18c3f625b87fba04e4afd"} Oct 30 20:26:13.974159 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:13.974119 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:13.974313 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:13.974190 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:13.974374 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:13.974356 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:26:13.974451 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:13.974443 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs podName:d7a37dcb-96d5-4c31-a7d7-08f663036a86 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:21.974425499 +0000 UTC m=+18.158640610 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs") pod "network-metrics-daemon-8sd9b" (UID: "d7a37dcb-96d5-4c31-a7d7-08f663036a86") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:26:13.974868 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:13.974841 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:26:13.974938 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:13.974898 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret podName:95f11ca1-5d71-4135-bf16-0fe6c24706c0 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:21.974882641 +0000 UTC m=+18.159097742 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret") pod "global-pull-secret-syncer-lwk8k" (UID: "95f11ca1-5d71-4135-bf16-0fe6c24706c0") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:26:14.075890 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:14.075577 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbs9\" (UniqueName: \"kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9\") pod \"network-check-target-tw7jg\" (UID: \"0001caf2-fffa-457e-8ab5-9574e11aced5\") " pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:14.075890 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:14.075757 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:26:14.075890 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:14.075777 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:26:14.075890 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:14.075791 2572 projected.go:196] Error preparing data for projected volume kube-api-access-wgbs9 for pod openshift-network-diagnostics/network-check-target-tw7jg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:26:14.075890 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:14.075850 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9 podName:0001caf2-fffa-457e-8ab5-9574e11aced5 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:22.075833101 +0000 UTC m=+18.260048204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wgbs9" (UniqueName: "kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9") pod "network-check-target-tw7jg" (UID: "0001caf2-fffa-457e-8ab5-9574e11aced5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:26:14.315458 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:14.314443 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:26:14.316495 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:14.315854 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:14.316495 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:14.315955 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:14.316495 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:14.316290 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:14.316495 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:14.316373 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:14.316495 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:14.316488 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:14.316823 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:14.316604 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:14.321153 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:14.319360 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:26:14.946327 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:14.946302 2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:26:15.250872 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:15.250790 2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:26:14.946321994Z","UUID":"46fb2124-3eb0-4752-bd85-2a123bea4428","Handler":null,"Name":"","Endpoint":""} Oct 30 20:26:15.253292 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:15.253273 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 20:26:15.253819 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:15.253302 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:26:15.399302 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:15.399262 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c4qhb" event={"ID":"46932d2f-65ad-4ac4-b422-f4991d35cbd2","Type":"ContainerStarted","Data":"16fc4d09d5d1f9437073ddc1a3da74fecbd43344bb549d29e7e9ee1be50ed1e0"} Oct 30 20:26:15.401964 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:15.401932 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" event={"ID":"6a305968-1cca-480f-9360-aeb99c0ef2de","Type":"ContainerStarted","Data":"b9b6a385893eb5fbed62d0887e36853dc5b111b3bc5a51786f781a0cb2891853"} Oct 30 20:26:15.403860 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:15.403833 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vbtnk" event={"ID":"4fe6a899-6382-488a-b52d-e20c68445e67","Type":"ContainerStarted","Data":"4f0a214e8be882b0dc4dc721ff757afc5b6daaadc26aef9883bdc8bacfdc6b6f"} Oct 30 20:26:15.405952 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:15.405929 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txlq" event={"ID":"0ddcb000-6271-4c42-a911-b3a4d14711c3","Type":"ContainerStarted","Data":"5f66023096e606c6c1637e314d74242d533281a945118e1bc153814e5e8638db"} Oct 30 20:26:15.515184 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:15.514932 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-c4qhb" podStartSLOduration=3.6255931820000002 podStartE2EDuration="11.514915148s" totalImagesPullingTime="7.889321966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:15.514890781 +0000 UTC m=+11.699105890" watchObservedRunningTime="2026-10-30 20:26:15.514915148 +0000 UTC m=+11.699130258" Oct 30 20:26:15.628198 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:15.628149 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-vbtnk" podStartSLOduration=3.602524835 podStartE2EDuration="11.62813662s" totalImagesPullingTime="8.025611785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:15.627694086 +0000 UTC m=+11.811909195" watchObservedRunningTime="2026-10-30 20:26:15.62813662 +0000 UTC m=+11.812351727" Oct 30 20:26:16.316410 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:16.315970 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:16.316410 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:16.316013 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:16.316410 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:16.315980 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:16.316410 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:16.316132 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:16.316410 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:16.316198 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:16.316410 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:16.316287 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:16.412255 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:16.412223 2572 generic.go:356] "Generic (PLEG): container finished" podID="0ddcb000-6271-4c42-a911-b3a4d14711c3" containerID="5f66023096e606c6c1637e314d74242d533281a945118e1bc153814e5e8638db" exitCode=0 Oct 30 20:26:16.412460 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:16.412301 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txlq" event={"ID":"0ddcb000-6271-4c42-a911-b3a4d14711c3","Type":"ContainerDied","Data":"5f66023096e606c6c1637e314d74242d533281a945118e1bc153814e5e8638db"} Oct 30 20:26:16.415102 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:16.415079 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9mq4r" event={"ID":"590e650a-39b7-4696-8bf9-c105a87fb012","Type":"ContainerStarted","Data":"36a1d586c8dc3cd5f4e4a31a349add15704e7997957eeb07e5be8961c8b12e7a"} Oct 30 20:26:17.418250 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:17.418217 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wwknf" event={"ID":"e0ac6e99-4947-4703-a31d-091a6cc9b9d7","Type":"ContainerStarted","Data":"e9583753af902604371e7f777d2896f09246f5fce15a38bfbfbbc9f5a5d62b68"} Oct 30 20:26:17.422870 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:17.422809 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" event={"ID":"6a305968-1cca-480f-9360-aeb99c0ef2de","Type":"ContainerStarted","Data":"836bdf6a4c433fc859d9e37a0c85bdc7e6e41151a10b11ee3db694adbc3de3b4"} Oct 30 20:26:17.534265 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:17.534221 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9mq4r" podStartSLOduration=5.159428778 podStartE2EDuration="13.534206837s" totalImagesPullingTime="8.374778059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:16.674071507 +0000 UTC m=+12.858286617" watchObservedRunningTime="2026-10-30 20:26:17.534206837 +0000 UTC m=+13.718421946" Oct 30 20:26:17.534564 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:17.534532 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wwknf" podStartSLOduration=3.280288258 podStartE2EDuration="13.534522679s" totalImagesPullingTime="10.254234421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:17.534130836 +0000 UTC m=+13.718345943" watchObservedRunningTime="2026-10-30 20:26:17.534522679 +0000 UTC m=+13.718737857" Oct 30 20:26:17.657983 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:17.657924 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" podStartSLOduration=4.32935177 podStartE2EDuration="13.657904625s" totalImagesPullingTime="9.328552855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:26:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:17.657533336 +0000 UTC m=+13.841748469" watchObservedRunningTime="2026-10-30 20:26:17.657904625 +0000 UTC m=+13.842119734" Oct 30 20:26:18.315816 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:18.315786 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:18.316017 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:18.315817 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:18.316017 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:18.315787 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:18.317254 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:18.317222 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:18.317408 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:18.317370 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:18.318711 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:18.318680 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:18.426727 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:18.426696 2572 generic.go:356] "Generic (PLEG): container finished" podID="0ddcb000-6271-4c42-a911-b3a4d14711c3" containerID="02c50a44279ea62be01cc7b9a9300c323ef894387058bbf8c5f8df0feec20d89" exitCode=0 Oct 30 20:26:18.427317 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:18.426796 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txlq" event={"ID":"0ddcb000-6271-4c42-a911-b3a4d14711c3","Type":"ContainerDied","Data":"02c50a44279ea62be01cc7b9a9300c323ef894387058bbf8c5f8df0feec20d89"} Oct 30 20:26:20.315556 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:20.315344 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:20.316038 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:20.315344 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:20.316038 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:20.315661 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:20.316038 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:20.315410 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:20.316038 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:20.315771 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:20.316038 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:20.315804 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:20.434095 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:20.434059 2572 generic.go:356] "Generic (PLEG): container finished" podID="0ddcb000-6271-4c42-a911-b3a4d14711c3" containerID="0be7fd9c458c8058cbc2afca64fde921fb26d3f8d4e643101238e37847a1af72" exitCode=0 Oct 30 20:26:20.434258 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:20.434116 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txlq" event={"ID":"0ddcb000-6271-4c42-a911-b3a4d14711c3","Type":"ContainerDied","Data":"0be7fd9c458c8058cbc2afca64fde921fb26d3f8d4e643101238e37847a1af72"} Oct 30 20:26:22.034580 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:22.034340 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:22.034958 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:22.034517 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:26:22.034958 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:22.034707 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret podName:95f11ca1-5d71-4135-bf16-0fe6c24706c0 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:38.034674024 +0000 UTC m=+34.218889130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret") pod "global-pull-secret-syncer-lwk8k" (UID: "95f11ca1-5d71-4135-bf16-0fe6c24706c0") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:26:22.034958 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:22.034707 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:26:22.034958 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:22.034762 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs podName:d7a37dcb-96d5-4c31-a7d7-08f663036a86 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:38.034746606 +0000 UTC m=+34.218961692 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs") pod "network-metrics-daemon-8sd9b" (UID: "d7a37dcb-96d5-4c31-a7d7-08f663036a86") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:26:22.034958 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:22.034621 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:22.135156 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:22.135127 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbs9\" (UniqueName: \"kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9\") pod \"network-check-target-tw7jg\" (UID: \"0001caf2-fffa-457e-8ab5-9574e11aced5\") " pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:22.135291 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:22.135276 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:26:22.135345 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:22.135297 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:26:22.135345 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:22.135311 2572 projected.go:196] Error preparing data for projected volume kube-api-access-wgbs9 for pod openshift-network-diagnostics/network-check-target-tw7jg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:26:22.135476 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:22.135367 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9 podName:0001caf2-fffa-457e-8ab5-9574e11aced5 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:38.135348264 +0000 UTC m=+34.319563364 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wgbs9" (UniqueName: "kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9") pod "network-check-target-tw7jg" (UID: "0001caf2-fffa-457e-8ab5-9574e11aced5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:26:22.315240 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:22.315155 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:22.315240 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:22.315187 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:22.315432 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:22.315275 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:22.315432 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:22.315323 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:22.315512 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:22.315439 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:22.315512 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:22.315476 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:22.442214 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:22.442182 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" event={"ID":"19997935-7a8e-4747-bb75-4532f1a72db5","Type":"ContainerStarted","Data":"470e7e48ab8ccfb86762acebe046194fcda4f6629c058cd9713facd69e85de67"} Oct 30 20:26:22.442370 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:22.442259 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" event={"ID":"19997935-7a8e-4747-bb75-4532f1a72db5","Type":"ContainerStarted","Data":"919afc6c5ec6974ced323fc5473a29d9154051c0e556f5cca5ccacd4e78053f5"} Oct 30 20:26:22.442370 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:22.442291 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" event={"ID":"19997935-7a8e-4747-bb75-4532f1a72db5","Type":"ContainerStarted","Data":"35e2bd938d84b8181e9fe890509d745acee8d28df62837c287c5bea5f5870506"} Oct 30 20:26:22.442370 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:22.442354 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" event={"ID":"19997935-7a8e-4747-bb75-4532f1a72db5","Type":"ContainerStarted","Data":"9df829f42333cc32c0883075c8c4417dabf715aa4aea6ba8077742ac1d42d90a"} Oct 30 20:26:22.442569 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:22.442397 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" event={"ID":"19997935-7a8e-4747-bb75-4532f1a72db5","Type":"ContainerStarted","Data":"b92f319eeed70f21489e19bac7c1e3a0440c92b57a71f0e6de7c6e41c04d6058"} Oct 30 20:26:22.442569 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:22.442430 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" event={"ID":"19997935-7a8e-4747-bb75-4532f1a72db5","Type":"ContainerStarted","Data":"98501eb3f414ad2c8aa36e5512aed03071eabe25c4d0ea2adf751a3fdf12029f"} Oct 30 20:26:23.447053 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:23.447017 2572 generic.go:356] "Generic (PLEG): container finished" podID="0ddcb000-6271-4c42-a911-b3a4d14711c3" containerID="00a526f80246081311fffc5e548133db38dd5f47066079b2c06b582510a7c7c6" exitCode=0 Oct 30 20:26:23.447584 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:23.447083 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txlq" event={"ID":"0ddcb000-6271-4c42-a911-b3a4d14711c3","Type":"ContainerDied","Data":"00a526f80246081311fffc5e548133db38dd5f47066079b2c06b582510a7c7c6"} Oct 30 20:26:24.316303 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:24.316266 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:24.316465 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:24.316358 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:24.316465 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:24.316419 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:24.316465 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:24.316409 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:24.316566 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:24.316490 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:24.316566 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:24.316545 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:24.451799 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:24.451714 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" event={"ID":"19997935-7a8e-4747-bb75-4532f1a72db5","Type":"ContainerStarted","Data":"ebbf80ca475157e437bd07e3f1923a53d2822437cdd1d2428be6e53c6aba53be"} Oct 30 20:26:24.454445 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:24.454421 2572 generic.go:356] "Generic (PLEG): container finished" podID="0ddcb000-6271-4c42-a911-b3a4d14711c3" containerID="152bbfcd3373a41f12ab164d5c5d03a85b2d039fc31541e7ee344b8f293ceaf1" exitCode=0 Oct 30 20:26:24.454580 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:24.454462 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txlq" event={"ID":"0ddcb000-6271-4c42-a911-b3a4d14711c3","Type":"ContainerDied","Data":"152bbfcd3373a41f12ab164d5c5d03a85b2d039fc31541e7ee344b8f293ceaf1"} Oct 30 20:26:25.459744 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:25.459710 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txlq" event={"ID":"0ddcb000-6271-4c42-a911-b3a4d14711c3","Type":"ContainerStarted","Data":"05509ca82486d1e7e6ec0bf3a1c7525a43be32e43c386636f4e8ab139a2e6b17"} Oct 30 20:26:25.589058 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:25.589004 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6txlq" podStartSLOduration=9.327535585 podStartE2EDuration="21.588989344s" totalImagesPullingTime="12.261453759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:26:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:25.588593692 +0000 UTC m=+21.772808800" watchObservedRunningTime="2026-10-30 20:26:25.588989344 +0000 UTC m=+21.773204451" Oct 30 20:26:26.315362 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:26.315336 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:26.315498 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:26.315342 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:26.315498 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:26.315462 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:26.315498 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:26.315482 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:26.315652 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:26.315608 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:26.315699 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:26.315687 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:26.466753 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:26.466534 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" event={"ID":"19997935-7a8e-4747-bb75-4532f1a72db5","Type":"ContainerStarted","Data":"0dcfaa7fea2b7471e4cd341d9f411029e55c82132927894a3ce54dfc3dea5f77"} Oct 30 20:26:26.740249 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:26.740125 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" podStartSLOduration=7.582351245 podStartE2EDuration="22.740111912s" totalImagesPullingTime="15.157760667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:26.7395569 +0000 UTC m=+22.923772008" watchObservedRunningTime="2026-10-30 20:26:26.740111912 +0000 UTC m=+22.924327064" Oct 30 20:26:27.469340 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:27.469311 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:27.469768 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:27.469418 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:27.469768 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:27.469467 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:27.487224 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:27.487192 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:27.488317 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:27.488297 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:28.315898 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:28.315866 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:28.315898 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:28.315886 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:28.316116 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:28.316031 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:28.316116 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:28.316068 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:28.316221 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:28.316155 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:28.316221 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:28.316211 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:28.975982 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:28.975912 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kfsl_dff6e046-945d-4553-ba94-908dbb1a0c23/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:29.125816 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:29.125774 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8sd9b"] Oct 30 20:26:29.125997 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:29.125924 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:29.126070 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:29.126042 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:29.129883 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:29.129857 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lwk8k"] Oct 30 20:26:29.130046 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:29.129952 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:29.130046 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:29.130000 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tw7jg"] Oct 30 20:26:29.130046 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:29.130035 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:29.130210 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:29.130081 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:29.130210 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:29.130175 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:29.240200 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:29.240115 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kfsl_dff6e046-945d-4553-ba94-908dbb1a0c23/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:29.755851 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:29.755828 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vbtnk_4fe6a899-6382-488a-b52d-e20c68445e67/dns-node-resolver/0.log" Oct 30 20:26:29.847262 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:29.847230 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:26:29.847466 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:29.847413 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:26:29.847852 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:29.847828 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:26:30.315996 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:30.315966 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:30.316537 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:30.315978 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:30.316537 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:30.316096 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:30.316537 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:30.316173 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:30.688488 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:30.688416 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zttcb_10982725-01b4-4d8e-8ad4-44d43608e6b0/node-ca/0.log" Oct 30 20:26:31.315995 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:31.315963 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:31.316160 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:31.316085 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:32.315739 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:32.315707 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:32.315739 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:32.315747 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:32.315953 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:32.315861 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:32.316003 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:32.315980 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:33.315932 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:33.315901 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:33.316319 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:33.316010 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:34.315942 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:34.315905 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:34.316318 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:34.315998 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:34.316318 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:34.316086 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:34.316318 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:34.316192 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:35.315541 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:35.315504 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:35.315708 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:35.315617 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:36.315265 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:36.315229 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:36.315730 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:36.315341 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:36.315730 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:36.315412 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:36.315730 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:36.315509 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:37.315922 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:37.315888 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:37.316411 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:37.316007 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:38.042372 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:38.042331 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:38.042372 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:38.042377 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:38.042615 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:38.042488 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:26:38.042615 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:38.042487 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:26:38.042615 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:38.042544 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret podName:95f11ca1-5d71-4135-bf16-0fe6c24706c0 nodeName:}" failed. No retries permitted until 2026-10-30 20:27:10.042530238 +0000 UTC m=+66.226745341 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret") pod "global-pull-secret-syncer-lwk8k" (UID: "95f11ca1-5d71-4135-bf16-0fe6c24706c0") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:26:38.042615 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:38.042557 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs podName:d7a37dcb-96d5-4c31-a7d7-08f663036a86 nodeName:}" failed. No retries permitted until 2026-10-30 20:27:10.042551285 +0000 UTC m=+66.226766370 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs") pod "network-metrics-daemon-8sd9b" (UID: "d7a37dcb-96d5-4c31-a7d7-08f663036a86") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:26:38.142955 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:38.142921 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbs9\" (UniqueName: \"kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9\") pod \"network-check-target-tw7jg\" (UID: \"0001caf2-fffa-457e-8ab5-9574e11aced5\") " pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:38.143111 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:38.143054 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:26:38.143111 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:38.143070 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:26:38.143111 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:38.143080 2572 projected.go:196] Error preparing data for projected volume kube-api-access-wgbs9 for pod openshift-network-diagnostics/network-check-target-tw7jg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:26:38.143208 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:38.143127 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9 podName:0001caf2-fffa-457e-8ab5-9574e11aced5 nodeName:}" failed. No retries permitted until 2026-10-30 20:27:10.14311619 +0000 UTC m=+66.327331277 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wgbs9" (UniqueName: "kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9") pod "network-check-target-tw7jg" (UID: "0001caf2-fffa-457e-8ab5-9574e11aced5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:26:38.315744 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:38.315649 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:38.315915 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:38.315769 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:38.315915 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:38.315819 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:38.316375 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:38.315918 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:39.316003 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:39.315971 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:39.316413 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:39.316076 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:40.315295 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:40.315266 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:40.315295 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:40.315304 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:40.315529 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:40.315415 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:40.315575 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:40.315545 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:41.315518 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:41.315479 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:41.315968 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:41.315617 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:42.315915 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:42.315881 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:42.316308 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:42.315883 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:42.316308 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:42.315995 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:42.316308 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:42.316075 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:43.315871 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:43.315832 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:43.316045 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:43.315946 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:44.316875 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:44.316838 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:44.317270 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:44.316953 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:44.317270 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:44.316995 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:44.317270 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:44.317085 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:45.315655 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:45.315619 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:45.315852 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:45.315774 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:46.315114 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:46.315072 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:46.315114 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:46.315117 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:46.315559 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:46.315234 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:46.315559 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:46.315270 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:47.315572 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:47.315530 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:47.315976 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:47.315657 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sd9b" podUID="d7a37dcb-96d5-4c31-a7d7-08f663036a86" Oct 30 20:26:48.316071 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.316029 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:48.316071 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.316066 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:48.316583 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:48.316147 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw7jg" podUID="0001caf2-fffa-457e-8ab5-9574e11aced5" Oct 30 20:26:48.316583 ip-10-0-139-195 kubenswrapper[2572]: E1030 20:26:48.316270 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwk8k" podUID="95f11ca1-5d71-4135-bf16-0fe6c24706c0" Oct 30 20:26:48.551640 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.551612 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeReady" Oct 30 20:26:48.551798 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.551747 2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 20:26:48.718366 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.718334 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9hkvc"] Oct 30 20:26:48.720104 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.720090 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6h6z6"] Oct 30 20:26:48.720231 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.720214 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:48.721521 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.721502 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vgwpl"] Oct 30 20:26:48.721660 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.721611 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6h6z6" Oct 30 20:26:48.723010 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.722993 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vgwpl" Oct 30 20:26:48.744855 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.744832 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6h6z6"] Oct 30 20:26:48.748009 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.747987 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9hkvc"] Oct 30 20:26:48.774775 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.774740 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vgwpl"] Oct 30 20:26:48.777447 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.777424 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 20:26:48.777612 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.777569 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 20:26:48.777676 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.777625 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bpqv7\"" Oct 30 20:26:48.777731 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.777706 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 20:26:48.777786 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.777747 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 20:26:48.810216 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.810130 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8411ba88-6e16-48e1-ab47-a33b4d5f0766-config-volume\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:26:48.810216 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.810167 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdc49\" (UniqueName: \"kubernetes.io/projected/ece532ab-396d-46f7-9de2-9e19563a5528-kube-api-access-pdc49\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:48.810216 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.810191 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ece532ab-396d-46f7-9de2-9e19563a5528-crio-socket\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:48.810216 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.810220 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cf2w\" (UniqueName: \"kubernetes.io/projected/8411ba88-6e16-48e1-ab47-a33b4d5f0766-kube-api-access-6cf2w\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:26:48.810500 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.810275 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ece532ab-396d-46f7-9de2-9e19563a5528-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:48.810500 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.810418 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ece532ab-396d-46f7-9de2-9e19563a5528-data-volume\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:48.810500 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.810445 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw58z\" (UniqueName: \"kubernetes.io/projected/1d508cdb-d418-4f6c-89c5-45e02e93f9d0-kube-api-access-jw58z\") pod \"ingress-canary-6h6z6\" (UID: \"1d508cdb-d418-4f6c-89c5-45e02e93f9d0\") " pod="openshift-ingress-canary/ingress-canary-6h6z6" Oct 30 20:26:48.810500 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.810472 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8411ba88-6e16-48e1-ab47-a33b4d5f0766-metrics-tls\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:26:48.810735 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.810588 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ece532ab-396d-46f7-9de2-9e19563a5528-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:48.810735 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.810637 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8411ba88-6e16-48e1-ab47-a33b4d5f0766-tmp-dir\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:26:48.810735 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.810682 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d508cdb-d418-4f6c-89c5-45e02e93f9d0-cert\") pod \"ingress-canary-6h6z6\" (UID: \"1d508cdb-d418-4f6c-89c5-45e02e93f9d0\") " pod="openshift-ingress-canary/ingress-canary-6h6z6" Oct 30 20:26:48.877463 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.877428 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 20:26:48.877642 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.877500 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 20:26:48.877642 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.877575 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 20:26:48.877642 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.877621 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 20:26:48.877642 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.877631 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 20:26:48.877801 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.877671 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lwgrd\"" Oct 30 20:26:48.877801 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.877693 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bxjhb\"" Oct 30 20:26:48.911450 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.911424 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cf2w\" (UniqueName: \"kubernetes.io/projected/8411ba88-6e16-48e1-ab47-a33b4d5f0766-kube-api-access-6cf2w\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:26:48.911504 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.911453 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ece532ab-396d-46f7-9de2-9e19563a5528-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:48.911504 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.911476 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ece532ab-396d-46f7-9de2-9e19563a5528-data-volume\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:48.911504 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.911492 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw58z\" (UniqueName: \"kubernetes.io/projected/1d508cdb-d418-4f6c-89c5-45e02e93f9d0-kube-api-access-jw58z\") pod \"ingress-canary-6h6z6\" (UID: \"1d508cdb-d418-4f6c-89c5-45e02e93f9d0\") " pod="openshift-ingress-canary/ingress-canary-6h6z6" Oct 30 20:26:48.911613 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.911551 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8411ba88-6e16-48e1-ab47-a33b4d5f0766-metrics-tls\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:26:48.911613 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.911595 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ece532ab-396d-46f7-9de2-9e19563a5528-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:48.912017 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.911726 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8411ba88-6e16-48e1-ab47-a33b4d5f0766-tmp-dir\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:26:48.912017 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.911782 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d508cdb-d418-4f6c-89c5-45e02e93f9d0-cert\") pod \"ingress-canary-6h6z6\" (UID: \"1d508cdb-d418-4f6c-89c5-45e02e93f9d0\") " pod="openshift-ingress-canary/ingress-canary-6h6z6" Oct 30 20:26:48.912017 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.911810 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8411ba88-6e16-48e1-ab47-a33b4d5f0766-config-volume\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:26:48.912017 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.911838 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdc49\" (UniqueName: \"kubernetes.io/projected/ece532ab-396d-46f7-9de2-9e19563a5528-kube-api-access-pdc49\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:48.912017 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.911862 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ece532ab-396d-46f7-9de2-9e19563a5528-data-volume\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:48.912017 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.911877 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ece532ab-396d-46f7-9de2-9e19563a5528-crio-socket\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:48.912017 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.911986 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ece532ab-396d-46f7-9de2-9e19563a5528-crio-socket\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:48.912339 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.912030 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8411ba88-6e16-48e1-ab47-a33b4d5f0766-tmp-dir\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:26:48.912339 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.912252 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ece532ab-396d-46f7-9de2-9e19563a5528-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:48.912430 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.912406 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8411ba88-6e16-48e1-ab47-a33b4d5f0766-config-volume\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:26:48.915444 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.915419 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8411ba88-6e16-48e1-ab47-a33b4d5f0766-metrics-tls\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:26:48.915537 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.915488 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ece532ab-396d-46f7-9de2-9e19563a5528-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:48.915708 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.915604 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d508cdb-d418-4f6c-89c5-45e02e93f9d0-cert\") pod \"ingress-canary-6h6z6\" (UID: \"1d508cdb-d418-4f6c-89c5-45e02e93f9d0\") " pod="openshift-ingress-canary/ingress-canary-6h6z6" Oct 30 20:26:48.985285 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.985258 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdc49\" (UniqueName: \"kubernetes.io/projected/ece532ab-396d-46f7-9de2-9e19563a5528-kube-api-access-pdc49\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:48.985462 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.985365 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cf2w\" (UniqueName: \"kubernetes.io/projected/8411ba88-6e16-48e1-ab47-a33b4d5f0766-kube-api-access-6cf2w\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:26:48.985512 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:48.985464 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw58z\" (UniqueName: \"kubernetes.io/projected/1d508cdb-d418-4f6c-89c5-45e02e93f9d0-kube-api-access-jw58z\") pod \"ingress-canary-6h6z6\" (UID: \"1d508cdb-d418-4f6c-89c5-45e02e93f9d0\") " pod="openshift-ingress-canary/ingress-canary-6h6z6" Oct 30 20:26:49.034212 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:49.034153 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:26:49.037697 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:49.037679 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6h6z6" Oct 30 20:26:49.043463 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:49.043440 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vgwpl" Oct 30 20:26:49.315707 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:49.315676 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:26:49.367790 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:49.367722 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 20:26:49.367790 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:49.367721 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-78mnk\"" Oct 30 20:26:49.471166 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:49.471140 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vgwpl"] Oct 30 20:26:49.473731 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:49.473704 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6h6z6"] Oct 30 20:26:49.474978 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:49.474926 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8411ba88_6e16_48e1_ab47_a33b4d5f0766.slice/crio-f6cd82820d0547eaa5d568e5aa00374e422051909ee4a4a338f7992765c27855 WatchSource:0}: Error finding container f6cd82820d0547eaa5d568e5aa00374e422051909ee4a4a338f7992765c27855: Status 404 returned error can't find the container with id f6cd82820d0547eaa5d568e5aa00374e422051909ee4a4a338f7992765c27855 Oct 30 20:26:49.477048 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:49.477031 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9hkvc"] Oct 30 20:26:49.516047 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:49.515996 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6h6z6" event={"ID":"1d508cdb-d418-4f6c-89c5-45e02e93f9d0","Type":"ContainerStarted","Data":"45bdf7a357596324cf935426759a332c7c9d969bc2a2c8ba7a682663f0c26967"} Oct 30 20:26:49.517516 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:49.517485 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vgwpl" event={"ID":"8411ba88-6e16-48e1-ab47-a33b4d5f0766","Type":"ContainerStarted","Data":"f6cd82820d0547eaa5d568e5aa00374e422051909ee4a4a338f7992765c27855"} Oct 30 20:26:49.523650 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:49.523613 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece532ab_396d_46f7_9de2_9e19563a5528.slice/crio-9570b49e052d07f7a47404631ce0b9342499670ec3e6d0f0c0f0efa117106b70 WatchSource:0}: Error finding container 9570b49e052d07f7a47404631ce0b9342499670ec3e6d0f0c0f0efa117106b70: Status 404 returned error can't find the container with id 9570b49e052d07f7a47404631ce0b9342499670ec3e6d0f0c0f0efa117106b70 Oct 30 20:26:50.315437 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:50.315405 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:26:50.315607 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:50.315583 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:26:50.369463 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:50.369422 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 20:26:50.369463 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:50.369427 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5tplx\"" Oct 30 20:26:50.369463 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:50.369458 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 20:26:50.369865 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:50.369491 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mkv2p\"" Oct 30 20:26:50.369865 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:50.369505 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 20:26:50.522660 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:50.522543 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9hkvc" event={"ID":"ece532ab-396d-46f7-9de2-9e19563a5528","Type":"ContainerStarted","Data":"478e99f00e3de4cae2c64d2f4f2e0b7b2ba2e3621f59fbc878161efb6d257074"} Oct 30 20:26:50.522660 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:50.522622 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9hkvc" event={"ID":"ece532ab-396d-46f7-9de2-9e19563a5528","Type":"ContainerStarted","Data":"9570b49e052d07f7a47404631ce0b9342499670ec3e6d0f0c0f0efa117106b70"} Oct 30 20:26:50.954896 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:50.954864 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q"] Oct 30 20:26:50.956857 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:50.956830 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q" Oct 30 20:26:51.011425 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.011335 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 20:26:51.011606 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.011581 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4bqw6\"" Oct 30 20:26:51.016873 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.016845 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q"] Oct 30 20:26:51.029764 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.029724 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3b1f827c-2d6d-4d1d-ac35-23324ff897fa-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5kk8q\" (UID: \"3b1f827c-2d6d-4d1d-ac35-23324ff897fa\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q" Oct 30 20:26:51.130581 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.130538 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3b1f827c-2d6d-4d1d-ac35-23324ff897fa-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5kk8q\" (UID: \"3b1f827c-2d6d-4d1d-ac35-23324ff897fa\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q" Oct 30 20:26:51.133791 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.133761 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3b1f827c-2d6d-4d1d-ac35-23324ff897fa-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5kk8q\" (UID: \"3b1f827c-2d6d-4d1d-ac35-23324ff897fa\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q" Oct 30 20:26:51.269455 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.269343 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q" Oct 30 20:26:51.425008 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.424974 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d5bb744b-ptq76"] Oct 30 20:26:51.427025 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.426999 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.485285 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.485255 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 20:26:51.485513 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.485499 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6w4b8\"" Oct 30 20:26:51.485577 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.485499 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 20:26:51.486032 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.486011 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 20:26:51.528475 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.528377 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9hkvc" event={"ID":"ece532ab-396d-46f7-9de2-9e19563a5528","Type":"ContainerStarted","Data":"9a7fcad97cb829c1945e85ad28e21afc815a1cdd4011ccb5c191af06d17cd465"} Oct 30 20:26:51.533004 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.532973 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-serving-cert\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.533156 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.533132 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-oauth-config\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.533208 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.533157 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-oauth-serving-cert\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.533208 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.533181 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc7c8\" (UniqueName: \"kubernetes.io/projected/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-kube-api-access-xc7c8\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.533285 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.533210 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-service-ca\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.533285 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.533230 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-tmp\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.533285 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.533255 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-config\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.533804 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.533773 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 20:26:51.533907 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.533899 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 20:26:51.533966 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.533930 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 20:26:51.534027 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.534012 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 20:26:51.535503 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.535479 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d5bb744b-ptq76"] Oct 30 20:26:51.633680 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.633642 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-config\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.633842 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.633690 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-serving-cert\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.633842 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.633789 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-oauth-config\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.633842 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.633812 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-oauth-serving-cert\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.633842 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.633838 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc7c8\" (UniqueName: \"kubernetes.io/projected/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-kube-api-access-xc7c8\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.634076 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.633897 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-service-ca\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.634076 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.633947 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-tmp\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.634366 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.634338 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-tmp\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.634552 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.634521 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-config\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.634635 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.634598 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-oauth-serving-cert\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.634803 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.634759 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-service-ca\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.636628 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.636609 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-oauth-config\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.636898 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.636879 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-serving-cert\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.657215 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.657184 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q"] Oct 30 20:26:51.661210 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:51.661185 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1f827c_2d6d_4d1d_ac35_23324ff897fa.slice/crio-85a7e16f66ac99d6a3befab3fb2fe7afe59b9000a12c79c93cd416f45e23518e WatchSource:0}: Error finding container 85a7e16f66ac99d6a3befab3fb2fe7afe59b9000a12c79c93cd416f45e23518e: Status 404 returned error can't find the container with id 85a7e16f66ac99d6a3befab3fb2fe7afe59b9000a12c79c93cd416f45e23518e Oct 30 20:26:51.693146 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.693109 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc7c8\" (UniqueName: \"kubernetes.io/projected/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-kube-api-access-xc7c8\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:51.740045 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:51.739998 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:26:52.174416 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:52.174323 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d5bb744b-ptq76"] Oct 30 20:26:52.178960 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:26:52.178911 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda459d678_44f2_4eed_9cb2_f370f1dcc8c1.slice/crio-fbe58304c85541f8664e66f2d8e258a24f94270321951fbbc1e91fb4125b3b83 WatchSource:0}: Error finding container fbe58304c85541f8664e66f2d8e258a24f94270321951fbbc1e91fb4125b3b83: Status 404 returned error can't find the container with id fbe58304c85541f8664e66f2d8e258a24f94270321951fbbc1e91fb4125b3b83 Oct 30 20:26:52.533166 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:52.533121 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5bb744b-ptq76" event={"ID":"a459d678-44f2-4eed-9cb2-f370f1dcc8c1","Type":"ContainerStarted","Data":"fbe58304c85541f8664e66f2d8e258a24f94270321951fbbc1e91fb4125b3b83"} Oct 30 20:26:52.534572 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:52.534542 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q" event={"ID":"3b1f827c-2d6d-4d1d-ac35-23324ff897fa","Type":"ContainerStarted","Data":"85a7e16f66ac99d6a3befab3fb2fe7afe59b9000a12c79c93cd416f45e23518e"} Oct 30 20:26:52.536715 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:52.536685 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vgwpl" event={"ID":"8411ba88-6e16-48e1-ab47-a33b4d5f0766","Type":"ContainerStarted","Data":"77f36aea63066072534833491a3c997d509a8e2f1ae0733b29474ac997ff23cf"} Oct 30 20:26:52.536830 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:52.536739 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vgwpl" event={"ID":"8411ba88-6e16-48e1-ab47-a33b4d5f0766","Type":"ContainerStarted","Data":"86978c82036eaab1a06c263a54c81fa86d6472a0d9453c2777477dd88a1e3161"} Oct 30 20:26:52.536885 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:52.536842 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vgwpl" Oct 30 20:26:53.543132 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:53.543083 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6h6z6" event={"ID":"1d508cdb-d418-4f6c-89c5-45e02e93f9d0","Type":"ContainerStarted","Data":"c51473e9fd8a45fa05f8ee4bc3c591483d2440db39616dcd895b48a74f0890d7"} Oct 30 20:26:53.545450 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:53.545419 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q" event={"ID":"3b1f827c-2d6d-4d1d-ac35-23324ff897fa","Type":"ContainerStarted","Data":"ab0eeac664ee5ba129a0f7420e13492e60d96f5b3daba0b3f1d0865097b9aa48"} Oct 30 20:26:53.545612 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:53.545470 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q" Oct 30 20:26:53.548215 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:53.548187 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9hkvc" event={"ID":"ece532ab-396d-46f7-9de2-9e19563a5528","Type":"ContainerStarted","Data":"896aa21448b43ad558433e7f6da66f24e2bb24567a605a05881c82dff0df95f4"} Oct 30 20:26:53.551662 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:53.551638 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q" Oct 30 20:26:53.657614 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:53.657558 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6h6z6" podStartSLOduration=2.177513183 podStartE2EDuration="5.657542003s" totalImagesPullingTime="3.48002882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:53.657273383 +0000 UTC m=+49.841488493" watchObservedRunningTime="2026-10-30 20:26:53.657542003 +0000 UTC m=+49.841757113" Oct 30 20:26:53.658288 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:53.658259 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-vgwpl" podStartSLOduration=3.355842483 podStartE2EDuration="5.658248438s" totalImagesPullingTime="2.302405955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:52.654173619 +0000 UTC m=+48.838388728" watchObservedRunningTime="2026-10-30 20:26:53.658248438 +0000 UTC m=+49.842463540" Oct 30 20:26:53.778204 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:53.778145 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q" podStartSLOduration=2.330024936 podStartE2EDuration="3.778127522s" totalImagesPullingTime="1.448102586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:53.777702416 +0000 UTC m=+49.961917550" watchObservedRunningTime="2026-10-30 20:26:53.778127522 +0000 UTC m=+49.962342632" Oct 30 20:26:53.897277 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:53.897174 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9hkvc" podStartSLOduration=2.744162783 podStartE2EDuration="5.897156697s" totalImagesPullingTime="3.152993914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:26:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:53.896510272 +0000 UTC m=+50.080725405" watchObservedRunningTime="2026-10-30 20:26:53.897156697 +0000 UTC m=+50.081371805" Oct 30 20:26:55.556570 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:55.556483 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5bb744b-ptq76" event={"ID":"a459d678-44f2-4eed-9cb2-f370f1dcc8c1","Type":"ContainerStarted","Data":"fd10c28cba2fb11e7d13d38f7182c7764e914f7aaf741ce3c05d0770b3893c59"} Oct 30 20:26:55.673160 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:55.673109 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7d5bb744b-ptq76" podStartSLOduration=1.585776284 podStartE2EDuration="4.673096536s" totalImagesPullingTime="3.087320252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:55.672279769 +0000 UTC m=+51.856494876" watchObservedRunningTime="2026-10-30 20:26:55.673096536 +0000 UTC m=+51.857311644" Oct 30 20:26:59.438567 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.438532 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-77c5f96b9d-ll5kd"] Oct 30 20:26:59.440507 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.440491 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.484762 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.484734 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:26:59.601108 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.601074 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-oauth-config\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.601108 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.601113 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-service-ca\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.601316 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.601266 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-tmp\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.601473 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.601451 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-config\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.601559 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.601487 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-oauth-serving-cert\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.601676 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.601642 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-trusted-ca-bundle\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.601676 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.601662 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klngr\" (UniqueName: \"kubernetes.io/projected/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-kube-api-access-klngr\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.601778 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.601695 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-serving-cert\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.642408 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.642355 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 20:26:59.643352 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.643333 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77c5f96b9d-ll5kd"] Oct 30 20:26:59.702540 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.702473 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-trusted-ca-bundle\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.702540 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.702504 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klngr\" (UniqueName: \"kubernetes.io/projected/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-kube-api-access-klngr\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.702540 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.702534 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-serving-cert\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.702746 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.702555 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-oauth-config\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.702746 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.702571 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-service-ca\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.702746 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.702596 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-tmp\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.702746 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.702648 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-config\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.702746 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.702669 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-oauth-serving-cert\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.703141 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.703108 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-tmp\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.703430 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.703379 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-service-ca\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.703563 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.703543 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-oauth-serving-cert\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.703611 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.703550 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-trusted-ca-bundle\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.703611 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.703586 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-config\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.705128 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.705102 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-oauth-config\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.705228 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.705213 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-serving-cert\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:26:59.802517 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:26:59.802484 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klngr\" (UniqueName: \"kubernetes.io/projected/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-kube-api-access-klngr\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:00.050602 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:00.050570 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:00.437948 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:00.437875 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77c5f96b9d-ll5kd"] Oct 30 20:27:00.440844 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:27:00.440819 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea8cfb3_a932_4689_8eda_f25dc44ca94a.slice/crio-4eff3f83181a243c24aabb7a7daeac647728e7bdfb76044e570a31316e826e1b WatchSource:0}: Error finding container 4eff3f83181a243c24aabb7a7daeac647728e7bdfb76044e570a31316e826e1b: Status 404 returned error can't find the container with id 4eff3f83181a243c24aabb7a7daeac647728e7bdfb76044e570a31316e826e1b Oct 30 20:27:00.575371 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:00.575339 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c5f96b9d-ll5kd" event={"ID":"5ea8cfb3-a932-4689-8eda-f25dc44ca94a","Type":"ContainerStarted","Data":"353f1b1f4cd3fc6e41ba880f4fa88877efb099ff32a6cf8ab1c293d2800d9786"} Oct 30 20:27:00.575557 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:00.575410 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c5f96b9d-ll5kd" event={"ID":"5ea8cfb3-a932-4689-8eda-f25dc44ca94a","Type":"ContainerStarted","Data":"4eff3f83181a243c24aabb7a7daeac647728e7bdfb76044e570a31316e826e1b"} Oct 30 20:27:00.691622 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:00.691529 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-77c5f96b9d-ll5kd" podStartSLOduration=1.691515278 podStartE2EDuration="1.691515278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:00.690861395 +0000 UTC m=+56.875076503" watchObservedRunningTime="2026-10-30 20:27:00.691515278 +0000 UTC m=+56.875730386" Oct 30 20:27:01.114035 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.113998 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf"] Oct 30 20:27:01.115961 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.115947 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:01.170122 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.170088 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 20:27:01.170299 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.170191 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 20:27:01.170299 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.170202 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-chbcw\"" Oct 30 20:27:01.170299 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.170284 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:27:01.170299 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.170297 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:27:01.170474 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.170353 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 20:27:01.177534 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.177504 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf"] Oct 30 20:27:01.215576 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.215541 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eb20209-5da1-4ef7-8625-b06c2b6493ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:01.215576 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.215575 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb20209-5da1-4ef7-8625-b06c2b6493ef-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:01.215810 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.215603 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzzc8\" (UniqueName: \"kubernetes.io/projected/3eb20209-5da1-4ef7-8625-b06c2b6493ef-kube-api-access-xzzc8\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:01.215810 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.215741 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3eb20209-5da1-4ef7-8625-b06c2b6493ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:01.225300 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.225276 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf"] Oct 30 20:27:01.227435 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.227415 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.240009 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.239986 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qm6mr"] Oct 30 20:27:01.241826 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.241811 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.287972 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.287889 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 20:27:01.288285 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.288220 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 20:27:01.288285 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.288239 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 20:27:01.288285 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.288279 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-94phh\"" Oct 30 20:27:01.311038 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.311009 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf"] Oct 30 20:27:01.316881 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.316853 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.317025 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.316886 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f019c85d-f532-4848-905b-b7e4480adba1-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.317025 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.316918 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eb20209-5da1-4ef7-8625-b06c2b6493ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:01.317137 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.317104 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb20209-5da1-4ef7-8625-b06c2b6493ef-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:01.317191 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.317150 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.317191 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.317183 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzzc8\" (UniqueName: \"kubernetes.io/projected/3eb20209-5da1-4ef7-8625-b06c2b6493ef-kube-api-access-xzzc8\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:01.317283 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.317264 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3eb20209-5da1-4ef7-8625-b06c2b6493ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:01.317338 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.317295 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f019c85d-f532-4848-905b-b7e4480adba1-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.317816 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.317342 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.317939 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.317832 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89kxn\" (UniqueName: \"kubernetes.io/projected/f019c85d-f532-4848-905b-b7e4480adba1-kube-api-access-89kxn\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.322783 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.318791 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb20209-5da1-4ef7-8625-b06c2b6493ef-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:01.322783 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.320258 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eb20209-5da1-4ef7-8625-b06c2b6493ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:01.323568 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.323547 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3eb20209-5da1-4ef7-8625-b06c2b6493ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:01.386201 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.386113 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 20:27:01.386830 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.386809 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 20:27:01.386934 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.386830 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 20:27:01.387091 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.387076 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nwr4g\"" Oct 30 20:27:01.395591 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.395564 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzzc8\" (UniqueName: \"kubernetes.io/projected/3eb20209-5da1-4ef7-8625-b06c2b6493ef-kube-api-access-xzzc8\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:01.418649 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.418607 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.418649 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.418652 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f019c85d-f532-4848-905b-b7e4480adba1-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.418868 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.418679 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/420b750b-2a29-4432-84f7-0d7709493a41-metrics-client-ca\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.418868 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.418704 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-tls\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.418868 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.418727 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-wtmp\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.418868 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.418752 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.418868 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.418774 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-accelerators-collector-config\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.419032 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.418939 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.419084 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.419060 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-root\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.419124 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.419117 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-sys\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.419163 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.419141 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f019c85d-f532-4848-905b-b7e4480adba1-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.419213 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.419168 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-textfile\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.419213 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.419203 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nn8g\" (UniqueName: \"kubernetes.io/projected/420b750b-2a29-4432-84f7-0d7709493a41-kube-api-access-4nn8g\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.419307 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.419222 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.419307 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.419238 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89kxn\" (UniqueName: \"kubernetes.io/projected/f019c85d-f532-4848-905b-b7e4480adba1-kube-api-access-89kxn\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.419475 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.419452 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f019c85d-f532-4848-905b-b7e4480adba1-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.419551 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.419452 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.419551 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.419530 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f019c85d-f532-4848-905b-b7e4480adba1-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.421258 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.421233 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.421537 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.421520 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.427469 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.427450 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:01.489191 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.489162 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89kxn\" (UniqueName: \"kubernetes.io/projected/f019c85d-f532-4848-905b-b7e4480adba1-kube-api-access-89kxn\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.520304 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.520270 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/420b750b-2a29-4432-84f7-0d7709493a41-metrics-client-ca\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.520632 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.520310 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-tls\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.520632 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.520335 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-wtmp\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.520632 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.520362 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-accelerators-collector-config\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.520632 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.520403 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.520873 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.520796 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-root\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.520966 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.520944 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-root\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.521083 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.521055 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-sys\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.521203 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.521089 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-textfile\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.521203 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.521092 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/420b750b-2a29-4432-84f7-0d7709493a41-metrics-client-ca\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.521203 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.521144 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nn8g\" (UniqueName: \"kubernetes.io/projected/420b750b-2a29-4432-84f7-0d7709493a41-kube-api-access-4nn8g\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.521203 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.521161 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-sys\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.521203 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.521168 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-wtmp\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.521489 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.521444 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-textfile\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.521679 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.521653 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-accelerators-collector-config\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.523104 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.523083 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.523200 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.523184 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-tls\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.538926 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.538901 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:01.617460 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.617429 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nn8g\" (UniqueName: \"kubernetes.io/projected/420b750b-2a29-4432-84f7-0d7709493a41-kube-api-access-4nn8g\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.740604 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.740568 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:01.740781 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.740659 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:01.745513 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.745489 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:01.798321 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.798292 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf"] Oct 30 20:27:01.802031 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:27:01.802001 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb20209_5da1_4ef7_8625_b06c2b6493ef.slice/crio-d72a74997b080c5ebfbe201aadb40f621cb3bd41f963e38b68ae2c3cdea79f84 WatchSource:0}: Error finding container d72a74997b080c5ebfbe201aadb40f621cb3bd41f963e38b68ae2c3cdea79f84: Status 404 returned error can't find the container with id d72a74997b080c5ebfbe201aadb40f621cb3bd41f963e38b68ae2c3cdea79f84 Oct 30 20:27:01.854033 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.854006 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:01.863313 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:27:01.863288 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420b750b_2a29_4432_84f7_0d7709493a41.slice/crio-4ed2b64f18b5b5b9da7c1c8d45f3c754b03835c8b12014432f6cb0ec361daf62 WatchSource:0}: Error finding container 4ed2b64f18b5b5b9da7c1c8d45f3c754b03835c8b12014432f6cb0ec361daf62: Status 404 returned error can't find the container with id 4ed2b64f18b5b5b9da7c1c8d45f3c754b03835c8b12014432f6cb0ec361daf62 Oct 30 20:27:01.922770 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:01.922744 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf"] Oct 30 20:27:01.925886 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:27:01.925860 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf019c85d_f532_4848_905b_b7e4480adba1.slice/crio-792ee3a0c0bf66cf265bc5e3ce143c98ada063b623d998319e6c6d73ce70a5ec WatchSource:0}: Error finding container 792ee3a0c0bf66cf265bc5e3ce143c98ada063b623d998319e6c6d73ce70a5ec: Status 404 returned error can't find the container with id 792ee3a0c0bf66cf265bc5e3ce143c98ada063b623d998319e6c6d73ce70a5ec Oct 30 20:27:02.551215 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:02.551184 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vgwpl" Oct 30 20:27:02.583540 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:02.583504 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" event={"ID":"f019c85d-f532-4848-905b-b7e4480adba1","Type":"ContainerStarted","Data":"792ee3a0c0bf66cf265bc5e3ce143c98ada063b623d998319e6c6d73ce70a5ec"} Oct 30 20:27:02.585226 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:02.585190 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" event={"ID":"3eb20209-5da1-4ef7-8625-b06c2b6493ef","Type":"ContainerStarted","Data":"73de3579dbadd16c929c994b087d3f24c20e13616d7a1295f36fc9d828b6d77a"} Oct 30 20:27:02.585346 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:02.585245 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" event={"ID":"3eb20209-5da1-4ef7-8625-b06c2b6493ef","Type":"ContainerStarted","Data":"7937b76d9b2c3d64ae583aa11321563d99925b6b7793b690ac5da8a0d2359cce"} Oct 30 20:27:02.585346 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:02.585273 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" event={"ID":"3eb20209-5da1-4ef7-8625-b06c2b6493ef","Type":"ContainerStarted","Data":"d72a74997b080c5ebfbe201aadb40f621cb3bd41f963e38b68ae2c3cdea79f84"} Oct 30 20:27:02.586347 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:02.586325 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm6mr" event={"ID":"420b750b-2a29-4432-84f7-0d7709493a41","Type":"ContainerStarted","Data":"4ed2b64f18b5b5b9da7c1c8d45f3c754b03835c8b12014432f6cb0ec361daf62"} Oct 30 20:27:02.590148 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:02.590131 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:03.539725 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.539693 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:27:03.549937 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.549821 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.594913 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.594874 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" event={"ID":"3eb20209-5da1-4ef7-8625-b06c2b6493ef","Type":"ContainerStarted","Data":"de9f5ed7ddff4b905376e3ee8eafcbf6be2268260aa116ebdf18acf87bfeab61"} Oct 30 20:27:03.596884 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.596852 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm6mr" event={"ID":"420b750b-2a29-4432-84f7-0d7709493a41","Type":"ContainerStarted","Data":"bc6a0881470d951a107994760dff7d53ec663177903a663e7ca9b6d242b62d8d"} Oct 30 20:27:03.604142 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.604078 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 20:27:03.604142 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.604093 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nnmlc\"" Oct 30 20:27:03.604142 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.604089 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 20:27:03.604142 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.604134 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 20:27:03.604355 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.604137 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 20:27:03.640954 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.640919 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.641132 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.640970 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.641132 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.641009 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.641132 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.641064 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-web-config\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.641250 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.641149 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd89k\" (UniqueName: \"kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-kube-api-access-hd89k\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.641250 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.641182 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.641250 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.641212 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.641364 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.641250 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.641364 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.641291 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-config-volume\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.641364 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.641308 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.641364 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.641340 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-config-out\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.641542 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.641447 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.641584 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.641543 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.702203 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.702162 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 20:27:03.702360 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.702218 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 20:27:03.702360 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.702222 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 20:27:03.702360 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.702266 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 20:27:03.702360 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.702301 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 20:27:03.703425 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.703380 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:27:03.742369 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.742344 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.742497 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.742397 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.742654 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.742635 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.742720 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.742676 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.742845 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.742718 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-web-config\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.742845 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.742797 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd89k\" (UniqueName: \"kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-kube-api-access-hd89k\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.743039 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.742848 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.743039 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.742882 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.743039 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.742916 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.743039 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.742976 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-config-volume\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.743039 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.743002 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.743278 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.743086 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-config-out\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.743278 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.743139 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.744240 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.743145 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.744240 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.743767 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.744240 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.743845 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.746230 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.746027 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.747056 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.747028 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.747615 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.747590 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.747941 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.747914 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.748483 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.748430 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-config-volume\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.748639 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.748610 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-web-config\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.749054 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.749001 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.749190 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.749174 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.750487 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.750465 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-config-out\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.801913 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.801881 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd89k\" (UniqueName: \"kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-kube-api-access-hd89k\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.875986 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.875957 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:03.962008 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:03.961952 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" podStartSLOduration=1.527879641 podStartE2EDuration="2.961935228s" totalImagesPullingTime="1.434055587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:03.961006693 +0000 UTC m=+60.145221804" watchObservedRunningTime="2026-10-30 20:27:03.961935228 +0000 UTC m=+60.146150336" Oct 30 20:27:04.308783 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:04.308742 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:27:04.312317 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:27:04.312286 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b04324_e2ac_435b_8756_3812700e9af7.slice/crio-a5002587a4732072cddd53b1bf39d63e1ad3aaf071fc57b426faf8c77402ec03 WatchSource:0}: Error finding container a5002587a4732072cddd53b1bf39d63e1ad3aaf071fc57b426faf8c77402ec03: Status 404 returned error can't find the container with id a5002587a4732072cddd53b1bf39d63e1ad3aaf071fc57b426faf8c77402ec03 Oct 30 20:27:04.601868 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:04.601769 2572 generic.go:356] "Generic (PLEG): container finished" podID="420b750b-2a29-4432-84f7-0d7709493a41" containerID="bc6a0881470d951a107994760dff7d53ec663177903a663e7ca9b6d242b62d8d" exitCode=0 Oct 30 20:27:04.601868 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:04.601857 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm6mr" event={"ID":"420b750b-2a29-4432-84f7-0d7709493a41","Type":"ContainerDied","Data":"bc6a0881470d951a107994760dff7d53ec663177903a663e7ca9b6d242b62d8d"} Oct 30 20:27:04.603620 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:04.603593 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9b04324-e2ac-435b-8756-3812700e9af7","Type":"ContainerStarted","Data":"a5002587a4732072cddd53b1bf39d63e1ad3aaf071fc57b426faf8c77402ec03"} Oct 30 20:27:04.605804 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:04.605781 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" event={"ID":"f019c85d-f532-4848-905b-b7e4480adba1","Type":"ContainerStarted","Data":"0f16c63f759ea29787b44211c034e780da650fa407ed0668a2034682943044f5"} Oct 30 20:27:04.605913 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:04.605843 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" event={"ID":"f019c85d-f532-4848-905b-b7e4480adba1","Type":"ContainerStarted","Data":"9b8ee7352afe975e692f5c3e05c2fdd1ecee0e643213cf6ab359214fc0755958"} Oct 30 20:27:04.605913 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:04.605874 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" event={"ID":"f019c85d-f532-4848-905b-b7e4480adba1","Type":"ContainerStarted","Data":"a0a2cc4f44650e59633a3d1e59725f614efb1f5647533ae0bce3ea2a0655cadf"} Oct 30 20:27:04.843608 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:04.843543 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" podStartSLOduration=2.093002173 podStartE2EDuration="3.843528224s" totalImagesPullingTime="1.750526051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:04.842667749 +0000 UTC m=+61.026882858" watchObservedRunningTime="2026-10-30 20:27:04.843528224 +0000 UTC m=+61.027743332" Oct 30 20:27:05.612513 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:05.611916 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm6mr" event={"ID":"420b750b-2a29-4432-84f7-0d7709493a41","Type":"ContainerStarted","Data":"ecb0990cbdb6ad571827b3c988095ac05743127e610423c1f90c41bda7fdb18c"} Oct 30 20:27:05.612513 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:05.611989 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm6mr" event={"ID":"420b750b-2a29-4432-84f7-0d7709493a41","Type":"ContainerStarted","Data":"9634fa9a37136728737d534a1254b8677615934bf8ecda27bf827851c6013696"} Oct 30 20:27:05.733490 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:05.733444 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qm6mr" podStartSLOduration=3.080326534 podStartE2EDuration="4.733427948s" totalImagesPullingTime="1.564889577s" totalInitContainerRuntime="88.211837ms" isStatefulPod=true podCreationTimestamp="2026-10-30 20:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:05.732319177 +0000 UTC m=+61.916534286" watchObservedRunningTime="2026-10-30 20:27:05.733427948 +0000 UTC m=+61.917643056" Oct 30 20:27:06.616507 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:06.616468 2572 generic.go:356] "Generic (PLEG): container finished" podID="f9b04324-e2ac-435b-8756-3812700e9af7" containerID="96f6550998f48971ac1ecf6f3bded5dd68338eec40b2f3f7df8c5567f1035af2" exitCode=0 Oct 30 20:27:06.616886 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:06.616554 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9b04324-e2ac-435b-8756-3812700e9af7","Type":"ContainerDied","Data":"96f6550998f48971ac1ecf6f3bded5dd68338eec40b2f3f7df8c5567f1035af2"} Oct 30 20:27:08.104878 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.104844 2572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77c5f96b9d-ll5kd"] Oct 30 20:27:08.154053 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.154021 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cf86b8865-9l5xr"] Oct 30 20:27:08.157399 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.157366 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.180979 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.180942 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44293dda-24ae-4aae-aa97-1e262c3186de-tmp\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.181139 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.180989 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4fnf\" (UniqueName: \"kubernetes.io/projected/44293dda-24ae-4aae-aa97-1e262c3186de-kube-api-access-w4fnf\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.181139 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.181068 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-trusted-ca-bundle\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.181139 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.181121 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-console-config\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.181272 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.181145 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-service-ca\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.181272 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.181221 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-serving-cert\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.181398 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.181368 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-oauth-serving-cert\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.181451 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.181417 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-oauth-config\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.217859 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.217779 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cf86b8865-9l5xr"] Oct 30 20:27:08.282636 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.282598 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-oauth-serving-cert\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.282800 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.282653 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-oauth-config\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.282800 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.282720 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44293dda-24ae-4aae-aa97-1e262c3186de-tmp\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.282800 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.282745 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4fnf\" (UniqueName: \"kubernetes.io/projected/44293dda-24ae-4aae-aa97-1e262c3186de-kube-api-access-w4fnf\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.282800 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.282769 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-trusted-ca-bundle\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.282977 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.282821 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-console-config\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.282977 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.282845 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-service-ca\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.282977 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.282871 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-serving-cert\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.283566 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.283426 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-oauth-serving-cert\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.283665 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.283642 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44293dda-24ae-4aae-aa97-1e262c3186de-tmp\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.283816 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.283793 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-trusted-ca-bundle\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.284236 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.284188 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-console-config\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.284236 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.284194 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-service-ca\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.285756 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.285568 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-oauth-config\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.285756 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.285716 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-serving-cert\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.344061 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.344029 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4fnf\" (UniqueName: \"kubernetes.io/projected/44293dda-24ae-4aae-aa97-1e262c3186de-kube-api-access-w4fnf\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.470114 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.470081 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:08.625693 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.625661 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9b04324-e2ac-435b-8756-3812700e9af7","Type":"ContainerStarted","Data":"b23bf95a50b4927953fb449472b426f4708434cbcc47cd8cf8ac35763f3f3a31"} Oct 30 20:27:08.625814 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.625752 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9b04324-e2ac-435b-8756-3812700e9af7","Type":"ContainerStarted","Data":"5c0d7597c7e82e7d2f0db173ad6ae4effb8be243d563b2b64aa1155a98aaa495"} Oct 30 20:27:08.850052 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:08.847466 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cf86b8865-9l5xr"] Oct 30 20:27:08.852884 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:27:08.852858 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44293dda_24ae_4aae_aa97_1e262c3186de.slice/crio-164bce9e38bd01c82a5295a3639118dbe978703f5028948ab9f98eb52616cb77 WatchSource:0}: Error finding container 164bce9e38bd01c82a5295a3639118dbe978703f5028948ab9f98eb52616cb77: Status 404 returned error can't find the container with id 164bce9e38bd01c82a5295a3639118dbe978703f5028948ab9f98eb52616cb77 Oct 30 20:27:09.632630 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:09.632598 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9b04324-e2ac-435b-8756-3812700e9af7","Type":"ContainerStarted","Data":"4e2ea3b15932e039201023f2710e7c35771606421e5173c96d40ce08a46a8a58"} Oct 30 20:27:09.633161 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:09.632667 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9b04324-e2ac-435b-8756-3812700e9af7","Type":"ContainerStarted","Data":"4ed423139336b82a159517f2298c1a45926f7e6997a830236453c194619d9e5c"} Oct 30 20:27:09.633161 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:09.632695 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9b04324-e2ac-435b-8756-3812700e9af7","Type":"ContainerStarted","Data":"45619b244dae9537e65d1c12498cac559619b243fb5f12ac0754454a4d154c89"} Oct 30 20:27:09.634191 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:09.634158 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf86b8865-9l5xr" event={"ID":"44293dda-24ae-4aae-aa97-1e262c3186de","Type":"ContainerStarted","Data":"f034529fd12dfcd14e0bb6d373ae32876cf03799f83e2fd25d38fafb0ccefd2b"} Oct 30 20:27:09.634292 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:09.634212 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf86b8865-9l5xr" event={"ID":"44293dda-24ae-4aae-aa97-1e262c3186de","Type":"ContainerStarted","Data":"164bce9e38bd01c82a5295a3639118dbe978703f5028948ab9f98eb52616cb77"} Oct 30 20:27:09.758098 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:09.758031 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6cf86b8865-9l5xr" podStartSLOduration=1.7580107489999999 podStartE2EDuration="1.758010749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:09.757853414 +0000 UTC m=+65.942068551" watchObservedRunningTime="2026-10-30 20:27:09.758010749 +0000 UTC m=+65.942225858" Oct 30 20:27:10.050809 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.050783 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:10.098340 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.098303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:27:10.098554 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.098349 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:27:10.149154 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.149119 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 20:27:10.149313 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.149197 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 20:27:10.161761 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.161726 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:27:10.161761 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.161741 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:27:10.198909 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.198871 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbs9\" (UniqueName: \"kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9\") pod \"network-check-target-tw7jg\" (UID: \"0001caf2-fffa-457e-8ab5-9574e11aced5\") " pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:27:10.249778 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.249697 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 20:27:10.310164 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.310132 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 20:27:10.322257 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.322234 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgbs9\" (UniqueName: \"kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9\") pod \"network-check-target-tw7jg\" (UID: \"0001caf2-fffa-457e-8ab5-9574e11aced5\") " pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:27:10.377634 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.377604 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-78mnk\"" Oct 30 20:27:10.387724 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.387700 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:27:10.477017 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.476990 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mkv2p\"" Oct 30 20:27:10.481862 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.481840 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5tplx\"" Oct 30 20:27:10.486973 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.486956 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:27:10.491951 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.491937 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:27:10.641873 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.641839 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9b04324-e2ac-435b-8756-3812700e9af7","Type":"ContainerStarted","Data":"02959c2b916e35e40644a616fd4bd315a1266f7b98d5f8b77e8bc1d8abbc370b"} Oct 30 20:27:10.763189 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.763102 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:27:10.767817 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.767792 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8sd9b"] Oct 30 20:27:10.767992 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.767935 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.806710 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.806671 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.806710 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.806708 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.806912 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.806791 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.806912 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.806816 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.806912 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.806832 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.806912 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.806850 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.806912 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.806865 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.807109 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.806925 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.807109 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.806997 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.807109 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.807032 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.807109 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.807053 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.807109 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.807076 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.807285 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.807142 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.807285 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.807164 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.807285 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.807179 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.807285 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.807194 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.807285 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.807261 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.807285 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.807284 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.807518 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.807299 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.807518 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.807317 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7g4d\" (UniqueName: \"kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-kube-api-access-f7g4d\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.816176 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.816129 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.360120362 podStartE2EDuration="7.816114095s" totalImagesPullingTime="4.44479693s" totalInitContainerRuntime="11.196803ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:10.814922322 +0000 UTC m=+66.999137430" watchObservedRunningTime="2026-10-30 20:27:10.816114095 +0000 UTC m=+67.000329202" Oct 30 20:27:10.817106 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:27:10.817080 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a37dcb_96d5_4c31_a7d7_08f663036a86.slice/crio-ea970a988820430430f16480ff40c16cc859236a183338f277d48931fe0302f3 WatchSource:0}: Error finding container ea970a988820430430f16480ff40c16cc859236a183338f277d48931fe0302f3: Status 404 returned error can't find the container with id ea970a988820430430f16480ff40c16cc859236a183338f277d48931fe0302f3 Oct 30 20:27:10.822249 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.822232 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 20:27:10.876437 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.876033 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 20:27:10.879013 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:27:10.878982 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f11ca1_5d71_4135_bf16_0fe6c24706c0.slice/crio-c5bbfc5789732b84c826edfc49bf665c799b1cb2ee44397c9ff2eb77998daba5 WatchSource:0}: Error finding container c5bbfc5789732b84c826edfc49bf665c799b1cb2ee44397c9ff2eb77998daba5: Status 404 returned error can't find the container with id c5bbfc5789732b84c826edfc49bf665c799b1cb2ee44397c9ff2eb77998daba5 Oct 30 20:27:10.881075 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:27:10.881050 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0001caf2_fffa_457e_8ab5_9574e11aced5.slice/crio-f7a612a206c05dbbc84b2330e873b6f105a49a7aa4c7cbb3de9fda66e9e124e6 WatchSource:0}: Error finding container f7a612a206c05dbbc84b2330e873b6f105a49a7aa4c7cbb3de9fda66e9e124e6: Status 404 returned error can't find the container with id f7a612a206c05dbbc84b2330e873b6f105a49a7aa4c7cbb3de9fda66e9e124e6 Oct 30 20:27:10.894738 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.894711 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 20:27:10.894738 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.894711 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 20:27:10.894913 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.894739 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 20:27:10.894913 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.894750 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 20:27:10.894913 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.894780 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 20:27:10.894913 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.894846 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 20:27:10.908680 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.908655 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.908781 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.908686 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.908781 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.908715 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.908878 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.908843 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.908932 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.908889 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.908932 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.908923 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.909029 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.908950 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.909029 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.908979 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.909029 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.909003 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.909230 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.909033 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.909230 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.909064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.909230 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.909099 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.909230 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.909135 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.909230 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.909174 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.909230 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.909201 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.909230 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.909227 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.909588 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.909257 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.909588 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.909281 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.909588 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.909296 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.909588 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.909343 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.909588 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.909420 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7g4d\" (UniqueName: \"kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-kube-api-access-f7g4d\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.910144 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.910122 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.910254 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.910190 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.912164 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.912135 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.912260 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.912135 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.912260 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.912229 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.912549 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.912526 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.912761 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.912744 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.913109 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.913092 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.913170 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.913154 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.913527 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.913507 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.913741 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.913725 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 20:27:10.913824 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.913725 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d9pbw\"" Oct 30 20:27:10.913824 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.913764 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fk349e28vqu0g\"" Oct 30 20:27:10.915308 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.915278 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:27:10.916420 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.916399 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lwk8k"] Oct 30 20:27:10.917489 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.917468 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tw7jg"] Oct 30 20:27:10.922364 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.922343 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.922532 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.922515 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.926195 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.926177 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 20:27:10.926282 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.926176 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 20:27:10.926282 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.926222 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 20:27:10.926505 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.926491 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 20:27:10.926708 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.926695 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 20:27:10.927314 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.927300 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 20:27:10.929609 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.929589 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.929880 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.929862 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.930088 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.930068 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.930231 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.930211 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.931767 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.931748 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.931946 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.931930 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:10.971979 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:10.971941 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7g4d\" (UniqueName: \"kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-kube-api-access-f7g4d\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:11.093065 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:11.092981 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:11.539737 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:11.539704 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:27:11.542574 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:27:11.542544 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f928cc6_d12f_42d6_82d3_4673ed8a9d07.slice/crio-6e618f27663d49d4bd8025e8fee1cc556fa1117ab56538d3db26cb14e57581d1 WatchSource:0}: Error finding container 6e618f27663d49d4bd8025e8fee1cc556fa1117ab56538d3db26cb14e57581d1: Status 404 returned error can't find the container with id 6e618f27663d49d4bd8025e8fee1cc556fa1117ab56538d3db26cb14e57581d1 Oct 30 20:27:11.650071 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:11.650038 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lwk8k" event={"ID":"95f11ca1-5d71-4135-bf16-0fe6c24706c0","Type":"ContainerStarted","Data":"c5bbfc5789732b84c826edfc49bf665c799b1cb2ee44397c9ff2eb77998daba5"} Oct 30 20:27:11.651404 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:11.651366 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8sd9b" event={"ID":"d7a37dcb-96d5-4c31-a7d7-08f663036a86","Type":"ContainerStarted","Data":"ea970a988820430430f16480ff40c16cc859236a183338f277d48931fe0302f3"} Oct 30 20:27:11.652781 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:11.652756 2572 generic.go:356] "Generic (PLEG): container finished" podID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerID="ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2" exitCode=0 Oct 30 20:27:11.652882 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:11.652837 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f928cc6-d12f-42d6-82d3-4673ed8a9d07","Type":"ContainerDied","Data":"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2"} Oct 30 20:27:11.652939 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:11.652900 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f928cc6-d12f-42d6-82d3-4673ed8a9d07","Type":"ContainerStarted","Data":"6e618f27663d49d4bd8025e8fee1cc556fa1117ab56538d3db26cb14e57581d1"} Oct 30 20:27:11.654035 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:11.654009 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tw7jg" event={"ID":"0001caf2-fffa-457e-8ab5-9574e11aced5","Type":"ContainerStarted","Data":"f7a612a206c05dbbc84b2330e873b6f105a49a7aa4c7cbb3de9fda66e9e124e6"} Oct 30 20:27:13.670201 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:13.669567 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8sd9b" event={"ID":"d7a37dcb-96d5-4c31-a7d7-08f663036a86","Type":"ContainerStarted","Data":"dd8fc6b5f119061d5b316de9cee77ad9f7c68243ac6f59d8ed65b443c03eee71"} Oct 30 20:27:13.670201 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:13.669680 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8sd9b" event={"ID":"d7a37dcb-96d5-4c31-a7d7-08f663036a86","Type":"ContainerStarted","Data":"8515a0dc0b8488f91fbb436585c340475e38346776d7828c82d601bf3f8ac683"} Oct 30 20:27:13.787942 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:13.787874 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8sd9b" podStartSLOduration=68.016143508 podStartE2EDuration="1m9.787854556s" totalImagesPullingTime="1.771711048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:13.786466476 +0000 UTC m=+69.970681585" watchObservedRunningTime="2026-10-30 20:27:13.787854556 +0000 UTC m=+69.972069668" Oct 30 20:27:15.682937 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.682703 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tw7jg" event={"ID":"0001caf2-fffa-457e-8ab5-9574e11aced5","Type":"ContainerStarted","Data":"2303f33ad9eb0eb444f3fb722fd6a8d8a5e22f0913045f1321a85c071bb66d4b"} Oct 30 20:27:15.682937 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.682873 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:27:15.707816 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.707776 2572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cf86b8865-9l5xr"] Oct 30 20:27:15.738098 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.738059 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-87bf75d78-kp5hd"] Oct 30 20:27:15.741715 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.741681 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.759407 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.759347 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-trusted-ca-bundle\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.759568 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.759415 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-service-ca\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.759568 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.759467 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-oauth-config\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.759568 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.759508 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-console-config\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.759568 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.759535 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-serving-cert\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.759568 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.759560 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0407316-563c-489c-9826-3c1372a6fbff-tmp\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.759827 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.759589 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fkx6\" (UniqueName: \"kubernetes.io/projected/e0407316-563c-489c-9826-3c1372a6fbff-kube-api-access-9fkx6\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.759827 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.759619 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-oauth-serving-cert\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.819401 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.819323 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tw7jg" podStartSLOduration=67.622004138 podStartE2EDuration="1m11.819303318s" totalImagesPullingTime="4.19729918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:15.799929857 +0000 UTC m=+71.984144990" watchObservedRunningTime="2026-10-30 20:27:15.819303318 +0000 UTC m=+72.003518426" Oct 30 20:27:15.819843 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.819810 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87bf75d78-kp5hd"] Oct 30 20:27:15.860889 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.860856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-oauth-config\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.861064 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.860896 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-console-config\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.861064 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.860924 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-serving-cert\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.861064 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.860948 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0407316-563c-489c-9826-3c1372a6fbff-tmp\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.861064 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.860977 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fkx6\" (UniqueName: \"kubernetes.io/projected/e0407316-563c-489c-9826-3c1372a6fbff-kube-api-access-9fkx6\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.861064 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.861005 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-oauth-serving-cert\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.861265 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.861070 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-trusted-ca-bundle\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.861265 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.861096 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-service-ca\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.861481 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.861440 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0407316-563c-489c-9826-3c1372a6fbff-tmp\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.861755 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.861730 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-console-config\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.861830 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.861759 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-service-ca\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.861971 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.861912 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-oauth-serving-cert\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.862142 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.862115 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-trusted-ca-bundle\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.864087 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.864059 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-oauth-config\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.864217 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.864195 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-serving-cert\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:15.922185 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:15.922125 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fkx6\" (UniqueName: \"kubernetes.io/projected/e0407316-563c-489c-9826-3c1372a6fbff-kube-api-access-9fkx6\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:16.055899 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:16.055692 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:16.446605 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:16.446527 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87bf75d78-kp5hd"] Oct 30 20:27:16.449142 ip-10-0-139-195 kubenswrapper[2572]: W1030 20:27:16.449110 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0407316_563c_489c_9826_3c1372a6fbff.slice/crio-9c7aa12704a9e14c288dd5a706e5ef45b8fe03363814b5a5f135fe87662138dd WatchSource:0}: Error finding container 9c7aa12704a9e14c288dd5a706e5ef45b8fe03363814b5a5f135fe87662138dd: Status 404 returned error can't find the container with id 9c7aa12704a9e14c288dd5a706e5ef45b8fe03363814b5a5f135fe87662138dd Oct 30 20:27:16.688740 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:16.688676 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f928cc6-d12f-42d6-82d3-4673ed8a9d07","Type":"ContainerStarted","Data":"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8"} Oct 30 20:27:16.689223 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:16.688763 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f928cc6-d12f-42d6-82d3-4673ed8a9d07","Type":"ContainerStarted","Data":"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447"} Oct 30 20:27:16.690301 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:16.690273 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87bf75d78-kp5hd" event={"ID":"e0407316-563c-489c-9826-3c1372a6fbff","Type":"ContainerStarted","Data":"1dcf8e094efe59f774bb8877908c8b744891c1161d6e126fd40460cf842221da"} Oct 30 20:27:16.690424 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:16.690322 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87bf75d78-kp5hd" event={"ID":"e0407316-563c-489c-9826-3c1372a6fbff","Type":"ContainerStarted","Data":"9c7aa12704a9e14c288dd5a706e5ef45b8fe03363814b5a5f135fe87662138dd"} Oct 30 20:27:16.809256 ip-10-0-139-195 kubenswrapper[2572]: I1030 20:27:16.809200 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-87bf75d78-kp5hd" podStartSLOduration=1.809183963 podStartE2EDuration="1.809183963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:16.808140287 +0000 UTC m=+72.992355400" watchObservedRunningTime="2026-10-30 20:27:16.809183963 +0000 UTC m=+72.993399072" Oct 30 20:27:17.009472 ip-10-0-139-195 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 20:27:17.025917 ip-10-0-139-195 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 20:27:17.026148 ip-10-0-139-195 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 20:27:17.026210 ip-10-0-139-195 systemd[1]: kubelet.service: Consumed 4.398s CPU time. Oct 30 20:27:17.038075 ip-10-0-139-195 systemd[1]: Starting Kubernetes Kubelet... Oct 30 20:27:17.106041 ip-10-0-139-195 kubenswrapper[5755]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:27:17.106041 ip-10-0-139-195 kubenswrapper[5755]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 20:27:17.106041 ip-10-0-139-195 kubenswrapper[5755]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:27:17.106041 ip-10-0-139-195 kubenswrapper[5755]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:27:17.108516 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108427 5755 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:27:17.108516 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108512 5755 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:27:17.108516 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108518 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108525 5755 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108532 5755 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108537 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108565 5755 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108570 5755 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108581 5755 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108585 5755 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108589 5755 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108594 5755 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108598 5755 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108602 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108606 5755 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108610 5755 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108614 5755 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108618 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108621 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108626 5755 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108630 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:27:17.108733 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108634 5755 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108638 5755 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108642 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108647 5755 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108651 5755 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108656 5755 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108660 5755 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108665 5755 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108669 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108673 5755 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108678 5755 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108682 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108687 5755 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108691 5755 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108695 5755 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108700 5755 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108704 5755 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108709 5755 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108714 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108719 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:27:17.109561 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108723 5755 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108727 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108732 5755 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108736 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108740 5755 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108745 5755 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108750 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108754 5755 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108758 5755 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108763 5755 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108767 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108771 5755 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108775 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108780 5755 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108784 5755 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108790 5755 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108794 5755 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108798 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108802 5755 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:27:17.110376 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108807 5755 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108811 5755 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108815 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108819 5755 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108825 5755 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108830 5755 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108834 5755 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108838 5755 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108843 5755 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108847 5755 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108851 5755 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108856 5755 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108861 5755 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108865 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108869 5755 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108873 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108877 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108882 5755 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108887 5755 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:27:17.111244 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108892 5755 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108897 5755 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108902 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108906 5755 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108910 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108915 5755 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108919 5755 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108923 5755 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108928 5755 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108934 5755 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108940 5755 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108945 5755 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108949 5755 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108954 5755 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108958 5755 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108963 5755 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108967 5755 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108971 5755 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108975 5755 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:27:17.112091 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108979 5755 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108983 5755 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108987 5755 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108992 5755 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.108996 5755 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.109000 5755 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.109004 5755 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.109017 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110729 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110743 5755 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110749 5755 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110755 5755 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110761 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110765 5755 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110770 5755 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110775 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110780 5755 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110785 5755 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110789 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110793 5755 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:27:17.112948 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110797 5755 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110801 5755 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110806 5755 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110810 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110814 5755 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110819 5755 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110823 5755 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110828 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110832 5755 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110836 5755 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110840 5755 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110844 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110848 5755 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110853 5755 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110857 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110861 5755 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110866 5755 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110870 5755 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110875 5755 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:27:17.113893 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110880 5755 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110886 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110890 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110894 5755 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110899 5755 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110905 5755 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110909 5755 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110913 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110918 5755 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110922 5755 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110929 5755 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110935 5755 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110940 5755 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110945 5755 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110949 5755 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110954 5755 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110958 5755 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110963 5755 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110967 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110972 5755 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:27:17.114774 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110976 5755 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110980 5755 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110984 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110988 5755 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110993 5755 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.110997 5755 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111001 5755 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111005 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111010 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111014 5755 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111018 5755 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111022 5755 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111026 5755 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111031 5755 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111036 5755 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111041 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111045 5755 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111050 5755 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111054 5755 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111058 5755 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:27:17.115576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111062 5755 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111066 5755 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111071 5755 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111074 5755 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111079 5755 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111084 5755 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111088 5755 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111095 5755 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111100 5755 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111107 5755 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111112 5755 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111116 5755 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111121 5755 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111125 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111129 5755 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111134 5755 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111138 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111142 5755 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111146 5755 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:27:17.116136 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111150 5755 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111154 5755 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111158 5755 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111162 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111166 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111171 5755 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111175 5755 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111182 5755 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111187 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111191 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111196 5755 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111200 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111204 5755 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111208 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111212 5755 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.111217 5755 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112110 5755 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112122 5755 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112129 5755 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:27:17.116635 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112134 5755 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112138 5755 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112143 5755 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112148 5755 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112152 5755 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112157 5755 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112162 5755 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112167 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112171 5755 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112175 5755 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112181 5755 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112187 5755 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112193 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112197 5755 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112201 5755 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112206 5755 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112210 5755 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112213 5755 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112218 5755 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:27:17.117096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112222 5755 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112231 5755 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112236 5755 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112241 5755 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112245 5755 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112250 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112254 5755 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112258 5755 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112263 5755 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112269 5755 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112276 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112282 5755 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112287 5755 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112292 5755 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112296 5755 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112301 5755 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112306 5755 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112310 5755 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112314 5755 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:27:17.117647 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112318 5755 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112322 5755 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112327 5755 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112331 5755 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112335 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112339 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112343 5755 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112347 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112352 5755 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112356 5755 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112360 5755 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112364 5755 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112368 5755 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112373 5755 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112379 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112402 5755 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112421 5755 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112426 5755 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112430 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112434 5755 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:27:17.118107 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112439 5755 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112443 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112447 5755 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112451 5755 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112455 5755 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112459 5755 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112464 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112468 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112474 5755 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112479 5755 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112483 5755 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112487 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112492 5755 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112496 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112501 5755 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112505 5755 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112509 5755 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112514 5755 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112518 5755 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112522 5755 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:27:17.118611 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112527 5755 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112531 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112535 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112539 5755 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112543 5755 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112547 5755 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112553 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112558 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112562 5755 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112567 5755 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112571 5755 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112575 5755 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112579 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112583 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112588 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112592 5755 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112597 5755 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112601 5755 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112605 5755 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:27:17.119258 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112609 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112615 5755 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112619 5755 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112623 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112629 5755 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.112634 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112777 5755 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112788 5755 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112798 5755 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112806 5755 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112813 5755 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112818 5755 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112825 5755 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112831 5755 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112837 5755 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112842 5755 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112848 5755 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112853 5755 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112858 5755 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112865 5755 flags.go:64] FLAG: --cgroup-root="" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112870 5755 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112875 5755 flags.go:64] FLAG: --client-ca-file="" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112880 5755 flags.go:64] FLAG: --cloud-provider="external" Oct 30 20:27:17.119959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112885 5755 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112892 5755 flags.go:64] FLAG: --cluster-domain="" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112897 5755 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112902 5755 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112907 5755 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112913 5755 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112919 5755 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112924 5755 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112930 5755 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112935 5755 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112942 5755 flags.go:64] FLAG: --contention-profiling="false" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112946 5755 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112951 5755 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112955 5755 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112960 5755 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112966 5755 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112971 5755 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112976 5755 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112980 5755 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112985 5755 flags.go:64] FLAG: --enable-server="true" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112990 5755 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.112997 5755 flags.go:64] FLAG: --event-burst="100" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113002 5755 flags.go:64] FLAG: --event-qps="50" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113007 5755 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 20:27:17.120593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113012 5755 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113016 5755 flags.go:64] FLAG: --eviction-hard="" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113023 5755 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113028 5755 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113036 5755 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113041 5755 flags.go:64] FLAG: --eviction-soft="" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113046 5755 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113051 5755 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113056 5755 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113062 5755 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113067 5755 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113071 5755 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113076 5755 flags.go:64] FLAG: --feature-gates="" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113083 5755 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113088 5755 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113093 5755 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113098 5755 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113102 5755 flags.go:64] FLAG: --healthz-port="10248" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113109 5755 flags.go:64] FLAG: --help="false" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113114 5755 flags.go:64] FLAG: --hostname-override="ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113120 5755 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113124 5755 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113129 5755 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 20:27:17.121163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113135 5755 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113140 5755 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113146 5755 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113150 5755 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113155 5755 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113160 5755 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113165 5755 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113170 5755 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113175 5755 flags.go:64] FLAG: --kube-reserved="" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113180 5755 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113184 5755 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113189 5755 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113194 5755 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113200 5755 flags.go:64] FLAG: --lock-file="" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113204 5755 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113209 5755 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113214 5755 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113223 5755 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113228 5755 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113234 5755 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113238 5755 flags.go:64] FLAG: --logging-format="text" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113243 5755 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113249 5755 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113253 5755 flags.go:64] FLAG: --manifest-url="" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113258 5755 flags.go:64] FLAG: --manifest-url-header="" Oct 30 20:27:17.121790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113265 5755 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113270 5755 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113278 5755 flags.go:64] FLAG: --max-pods="110" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113309 5755 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113317 5755 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113322 5755 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113328 5755 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113333 5755 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113338 5755 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113344 5755 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113365 5755 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113370 5755 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113375 5755 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113379 5755 flags.go:64] FLAG: --pod-cidr="" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113402 5755 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113407 5755 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113412 5755 flags.go:64] FLAG: --pods-per-core="0" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113416 5755 flags.go:64] FLAG: --port="10250" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113421 5755 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113426 5755 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-050d8981ac5e8b88f" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113433 5755 flags.go:64] FLAG: --qos-reserved="" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113438 5755 flags.go:64] FLAG: --read-only-port="10255" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113443 5755 flags.go:64] FLAG: --register-node="true" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113448 5755 flags.go:64] FLAG: --register-with-taints="" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113454 5755 flags.go:64] FLAG: --registry-burst="10" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113459 5755 flags.go:64] FLAG: --registry-qps="5" Oct 30 20:27:17.122461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113463 5755 flags.go:64] FLAG: --reserved-cpus="" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113470 5755 flags.go:64] FLAG: --reserved-memory="" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113476 5755 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113481 5755 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113486 5755 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113490 5755 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113495 5755 flags.go:64] FLAG: --runonce="false" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113500 5755 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113507 5755 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113512 5755 flags.go:64] FLAG: --seccomp-default="false" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113517 5755 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113522 5755 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113527 5755 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113532 5755 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113537 5755 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113541 5755 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113546 5755 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113551 5755 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113556 5755 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113561 5755 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113565 5755 flags.go:64] FLAG: --system-cgroups="" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113570 5755 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113579 5755 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113583 5755 flags.go:64] FLAG: --tls-cert-file="" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113588 5755 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 20:27:17.123288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113595 5755 flags.go:64] FLAG: --tls-min-version="" Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113602 5755 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113606 5755 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113611 5755 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113616 5755 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113621 5755 flags.go:64] FLAG: --v="2" Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113629 5755 flags.go:64] FLAG: --version="false" Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113636 5755 flags.go:64] FLAG: --vmodule="" Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113642 5755 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.113649 5755 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113869 5755 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113877 5755 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113882 5755 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113888 5755 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113893 5755 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113900 5755 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113904 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113908 5755 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113912 5755 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113918 5755 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113924 5755 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:27:17.123980 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113930 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113935 5755 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113939 5755 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113943 5755 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113947 5755 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113952 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113957 5755 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113961 5755 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113965 5755 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113969 5755 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113974 5755 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113978 5755 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113982 5755 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113986 5755 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113991 5755 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113995 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.113999 5755 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114003 5755 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114008 5755 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:27:17.124618 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114012 5755 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114016 5755 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114025 5755 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114030 5755 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114034 5755 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114038 5755 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114042 5755 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114048 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114052 5755 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114056 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114061 5755 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114065 5755 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114069 5755 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114073 5755 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114077 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114082 5755 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114086 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114090 5755 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114094 5755 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:27:17.125076 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114099 5755 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114103 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114107 5755 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114112 5755 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114117 5755 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114121 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114125 5755 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114129 5755 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114133 5755 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114137 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114141 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114145 5755 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114150 5755 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114154 5755 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114158 5755 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114162 5755 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114168 5755 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114172 5755 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114176 5755 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114180 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:27:17.125568 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114186 5755 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114190 5755 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114195 5755 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114199 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114204 5755 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114208 5755 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114212 5755 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114216 5755 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114220 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114224 5755 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114229 5755 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114233 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114237 5755 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114243 5755 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114247 5755 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114252 5755 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114256 5755 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114260 5755 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114265 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114269 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:27:17.126209 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114273 5755 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:27:17.126757 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114277 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:27:17.126757 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114281 5755 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:27:17.126757 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114285 5755 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:27:17.126757 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114289 5755 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:27:17.126757 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114294 5755 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:27:17.126757 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114298 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:27:17.126757 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114302 5755 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:27:17.126757 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114306 5755 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:27:17.126757 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114311 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:27:17.126757 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114315 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:27:17.126757 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114319 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:27:17.126757 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114326 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:27:17.126757 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114331 5755 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:27:17.126757 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114335 5755 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:27:17.126757 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114342 5755 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:27:17.126757 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114347 5755 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:27:17.127192 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.114409 5755 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:27:17.127192 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114466 5755 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:27:17.127192 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114474 5755 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:27:17.127192 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114479 5755 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:27:17.127192 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114483 5755 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:27:17.127192 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114488 5755 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:27:17.127192 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114492 5755 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:27:17.127192 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114496 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:27:17.127192 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114500 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:27:17.127192 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114505 5755 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:27:17.127192 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114509 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:27:17.127192 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114513 5755 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:27:17.127192 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114517 5755 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:27:17.127192 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114521 5755 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:27:17.127192 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114526 5755 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114530 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114534 5755 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114538 5755 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114542 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114547 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114551 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114554 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114558 5755 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114565 5755 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114572 5755 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114578 5755 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114583 5755 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114589 5755 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114593 5755 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114598 5755 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114602 5755 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114606 5755 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114611 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:27:17.127631 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114615 5755 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114619 5755 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114626 5755 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114632 5755 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114637 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114641 5755 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114645 5755 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114649 5755 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114654 5755 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114658 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114664 5755 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114668 5755 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114672 5755 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114676 5755 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114681 5755 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114684 5755 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114689 5755 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114693 5755 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114697 5755 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:27:17.128122 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114702 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114706 5755 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114710 5755 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114716 5755 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114720 5755 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114724 5755 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114729 5755 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114735 5755 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114739 5755 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114743 5755 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114748 5755 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114752 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114756 5755 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114761 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114765 5755 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114769 5755 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114773 5755 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114777 5755 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114781 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:27:17.128730 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114786 5755 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114790 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114794 5755 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114798 5755 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114802 5755 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114807 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114811 5755 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114816 5755 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114820 5755 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114825 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114829 5755 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114833 5755 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114838 5755 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114842 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114846 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114850 5755 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114856 5755 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114861 5755 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114865 5755 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114869 5755 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:27:17.129358 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114876 5755 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114881 5755 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114885 5755 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114889 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114893 5755 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114898 5755 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114903 5755 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114907 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114911 5755 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114915 5755 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114919 5755 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114924 5755 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114928 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114932 5755 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114938 5755 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.114942 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.120963 5755 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.120981 5755 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.121003 5755 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.121008 5755 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 20:27:17.129888 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121083 5755 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121089 5755 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121092 5755 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121095 5755 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121098 5755 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121100 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121103 5755 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121106 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121108 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121111 5755 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121113 5755 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121116 5755 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121119 5755 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121121 5755 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121124 5755 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121129 5755 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121133 5755 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121136 5755 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121138 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121141 5755 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:27:17.130447 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121144 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121146 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121149 5755 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121151 5755 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121154 5755 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121157 5755 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121160 5755 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121162 5755 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121165 5755 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121167 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121170 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121173 5755 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121175 5755 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121178 5755 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121181 5755 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121183 5755 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121186 5755 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121188 5755 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121191 5755 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:27:17.130981 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121194 5755 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121199 5755 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121203 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121207 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121210 5755 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121213 5755 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121216 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121220 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121223 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121225 5755 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121228 5755 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121230 5755 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121233 5755 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121235 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121238 5755 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121240 5755 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121243 5755 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121245 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121248 5755 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:27:17.131596 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121251 5755 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121254 5755 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121256 5755 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121259 5755 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121261 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121264 5755 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121266 5755 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121269 5755 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121271 5755 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121274 5755 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121276 5755 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121279 5755 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121281 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121283 5755 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121286 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121289 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121292 5755 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121294 5755 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121297 5755 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121299 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:27:17.132116 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121302 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121305 5755 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121308 5755 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121310 5755 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121313 5755 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121315 5755 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121318 5755 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121320 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121323 5755 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121325 5755 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121328 5755 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121331 5755 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121333 5755 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121336 5755 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121338 5755 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121341 5755 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121344 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121346 5755 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121360 5755 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121363 5755 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:27:17.132667 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121366 5755 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:27:17.133212 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121368 5755 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:27:17.133212 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121371 5755 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:27:17.133212 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121374 5755 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:27:17.133212 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121376 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:27:17.133212 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121379 5755 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:27:17.133212 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121396 5755 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:27:17.133212 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121401 5755 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:27:17.133212 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.121432 5755 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:27:17.133212 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121578 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:27:17.133212 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121582 5755 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:27:17.133212 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121585 5755 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:27:17.133212 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121588 5755 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:27:17.133212 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121591 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:27:17.133212 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121594 5755 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:27:17.133212 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121597 5755 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:27:17.133212 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121599 5755 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121601 5755 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121604 5755 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121606 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121609 5755 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121611 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121614 5755 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121617 5755 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121619 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121622 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121625 5755 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121628 5755 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121630 5755 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121632 5755 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121635 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121638 5755 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121640 5755 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121643 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121645 5755 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:27:17.133664 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121647 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121650 5755 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121652 5755 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121655 5755 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121657 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121660 5755 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121663 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121665 5755 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121669 5755 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121673 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121676 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121679 5755 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121682 5755 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121686 5755 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121688 5755 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121691 5755 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121693 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121696 5755 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:27:17.134398 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121698 5755 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121701 5755 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121703 5755 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121706 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121708 5755 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121711 5755 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121713 5755 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121716 5755 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121718 5755 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121720 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121723 5755 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121725 5755 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121728 5755 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121730 5755 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121732 5755 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121735 5755 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121737 5755 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121740 5755 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121742 5755 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121744 5755 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:27:17.135172 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121747 5755 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121749 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121754 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121757 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121760 5755 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121762 5755 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121765 5755 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121768 5755 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121770 5755 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121773 5755 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121775 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121778 5755 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121780 5755 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121783 5755 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121786 5755 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121788 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121790 5755 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121793 5755 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121796 5755 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121799 5755 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:27:17.135822 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121801 5755 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121804 5755 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121806 5755 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121809 5755 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121811 5755 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121814 5755 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121818 5755 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121822 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121824 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121827 5755 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121830 5755 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121832 5755 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121835 5755 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121837 5755 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121841 5755 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121843 5755 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121845 5755 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121848 5755 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121851 5755 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121853 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:27:17.136404 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121856 5755 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:27:17.137001 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121859 5755 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:27:17.137001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.121882 5755 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:27:17.137001 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121916 5755 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:27:17.137001 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121919 5755 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:27:17.137001 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121922 5755 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:27:17.137001 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121925 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:27:17.137001 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121927 5755 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:27:17.137001 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121930 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:27:17.137001 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121933 5755 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:27:17.137001 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121937 5755 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:27:17.137001 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121940 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:27:17.137001 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121943 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:27:17.137001 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121945 5755 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:27:17.137001 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121948 5755 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:27:17.137001 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121950 5755 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121953 5755 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121956 5755 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121958 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121961 5755 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121963 5755 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121966 5755 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121968 5755 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121971 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121973 5755 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121976 5755 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121978 5755 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121982 5755 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121985 5755 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121987 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121990 5755 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121992 5755 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121995 5755 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.121998 5755 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122001 5755 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:27:17.137468 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122003 5755 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122005 5755 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122008 5755 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122011 5755 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122013 5755 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122016 5755 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122018 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122020 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122023 5755 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122026 5755 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122028 5755 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122031 5755 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122033 5755 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122036 5755 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122038 5755 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122041 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122043 5755 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122046 5755 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122049 5755 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122051 5755 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:27:17.138048 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122053 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122056 5755 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122058 5755 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122061 5755 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122065 5755 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122068 5755 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122070 5755 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122073 5755 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122075 5755 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122077 5755 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122080 5755 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122084 5755 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122088 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122091 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122093 5755 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122096 5755 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122099 5755 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122103 5755 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122107 5755 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122111 5755 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:27:17.138621 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122114 5755 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122118 5755 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122122 5755 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122126 5755 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122130 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122134 5755 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122138 5755 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122142 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122145 5755 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122155 5755 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122159 5755 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122163 5755 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122167 5755 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122171 5755 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122175 5755 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122182 5755 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122187 5755 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122192 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122196 5755 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122200 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:27:17.139118 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122204 5755 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122208 5755 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122213 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122218 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122223 5755 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122228 5755 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122232 5755 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122236 5755 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122242 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122247 5755 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122251 5755 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122255 5755 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122261 5755 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:17.122264 5755 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.122345 5755 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.123408 5755 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.123473 5755 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.123650 5755 server.go:1014] "Starting client certificate rotation" Oct 30 20:27:17.139697 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.123800 5755 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:27:17.140137 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.123863 5755 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:21:04 +0000 UTC" deadline="2028-06-09 07:15:53.566600915 +0000 UTC" Oct 30 20:27:17.140137 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.123905 5755 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14098h48m36.442699632s" Oct 30 20:27:17.140137 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.124171 5755 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:27:17.140137 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.125304 5755 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:27:17.140137 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.126765 5755 log.go:25] "Validated CRI v1 runtime API" Oct 30 20:27:17.140137 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.128775 5755 log.go:25] "Validated CRI v1 image API" Oct 30 20:27:17.140137 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.129575 5755 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 20:27:17.140137 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.131860 5755 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c70833be-4520-4920-904c-a76540f204e5:/dev/nvme0n1p3 f5ce8070-923c-4036-9872-0a92e217993d:/dev/nvme0n1p4] Oct 30 20:27:17.140427 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.131882 5755 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13f03e50512a578a5ec3e178a77dcb47dc4637aa5fe2f0fe074b2e67172dba83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13f03e50512a578a5ec3e178a77dcb47dc4637aa5fe2f0fe074b2e67172dba83/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/164bce9e38bd01c82a5295a3639118dbe978703f5028948ab9f98eb52616cb77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/164bce9e38bd01c82a5295a3639118dbe978703f5028948ab9f98eb52616cb77/userdata/shm major:0 minor:367 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/365cb554216b8fa9574c9b4c326fc8927b9d610d05f3477e9fa04a4462aaf968/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/365cb554216b8fa9574c9b4c326fc8927b9d610d05f3477e9fa04a4462aaf968/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45bdf7a357596324cf935426759a332c7c9d969bc2a2c8ba7a682663f0c26967/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45bdf7a357596324cf935426759a332c7c9d969bc2a2c8ba7a682663f0c26967/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b9ee59e00f9f10151f45bc9eeda5290f733fa47f3e1c8e0aa5fde6fe999f48c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b9ee59e00f9f10151f45bc9eeda5290f733fa47f3e1c8e0aa5fde6fe999f48c/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4dbfa8cf88c5a111ad23499c0d035296b7b8e9a30edd98d6e13536a6b620899b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4dbfa8cf88c5a111ad23499c0d035296b7b8e9a30edd98d6e13536a6b620899b/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ed2b64f18b5b5b9da7c1c8d45f3c754b03835c8b12014432f6cb0ec361daf62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ed2b64f18b5b5b9da7c1c8d45f3c754b03835c8b12014432f6cb0ec361daf62/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4eff3f83181a243c24aabb7a7daeac647728e7bdfb76044e570a31316e826e1b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4eff3f83181a243c24aabb7a7daeac647728e7bdfb76044e570a31316e826e1b/userdata/shm major:0 minor:277 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5fd458ac66e32419632ae0455910180d935049c4b6fdbd6134b35228130793f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5fd458ac66e32419632ae0455910180d935049c4b6fdbd6134b35228130793f2/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e618f27663d49d4bd8025e8fee1cc556fa1117ab56538d3db26cb14e57581d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e618f27663d49d4bd8025e8fee1cc556fa1117ab56538d3db26cb14e57581d1/userdata/shm major:0 minor:435 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/792ee3a0c0bf66cf265bc5e3ce143c98ada063b623d998319e6c6d73ce70a5ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/792ee3a0c0bf66cf265bc5e3ce143c98ada063b623d998319e6c6d73ce70a5ec/userdata/shm major:0 minor:297 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81bd1aa7a4ea1b9b050060623a14ecd686bad9ce306fb444336a134c6c1c4b9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81bd1aa7a4ea1b9b050060623a14ecd686bad9ce306fb444336a134c6c1c4b9d/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85a7e16f66ac99d6a3befab3fb2fe7afe59b9000a12c79c93cd416f45e23518e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85a7e16f66ac99d6a3befab3fb2fe7afe59b9000a12c79c93cd416f45e23518e/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/879c830c59794efe325ec0e1c7671029cb1e7e5fdfd7af2736ab352f6eab8cba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/879c830c59794efe325ec0e1c7671029cb1e7e5fdfd7af2736ab352f6eab8cba/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/947e91e49a01c04511694a679764c49788f373f6bf7e637231fd11bc0f5f862b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/947e91e49a01c04511694a679764c49788f373f6bf7e637231fd11bc0f5f862b/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9570b49e052d07f7a47404631ce0b9342499670ec3e6d0f0c0f0efa117106b70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9570b49e052d07f7a47404631ce0b9342499670ec3e6d0f0c0f0efa117106b70/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c7aa12704a9e14c288dd5a706e5ef45b8fe03363814b5a5f135fe87662138dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c7aa12704a9e14c288dd5a706e5ef45b8fe03363814b5a5f135fe87662138dd/userdata/shm major:0 minor:471 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5002587a4732072cddd53b1bf39d63e1ad3aaf071fc57b426faf8c77402ec03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5002587a4732072cddd53b1bf39d63e1ad3aaf071fc57b426faf8c77402ec03/userdata/shm major:0 minor:351 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5bbfc5789732b84c826edfc49bf665c799b1cb2ee44397c9ff2eb77998daba5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5bbfc5789732b84c826edfc49bf665c799b1cb2ee44397c9ff2eb77998daba5/userdata/shm major:0 minor:415 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d72a74997b080c5ebfbe201aadb40f621cb3bd41f963e38b68ae2c3cdea79f84/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d72a74997b080c5ebfbe201aadb40f621cb3bd41f963e38b68ae2c3cdea79f84/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1b511941563444d88c34017d11fca27ce03fbfd82d708b2aff9eeb6c6c2f534/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1b511941563444d88c34017d11fca27ce03fbfd82d708b2aff9eeb6c6c2f534/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e78b32209ac052ac75dd366a4bfdbe197ee2de2c77c81bbc38acd54b4b9b3a75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e78b32209ac052ac75dd366a4bfdbe197ee2de2c77c81bbc38acd54b4b9b3a75/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea970a988820430430f16480ff40c16cc859236a183338f277d48931fe0302f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea970a988820430430f16480ff40c16cc859236a183338f277d48931fe0302f3/userdata/shm major:0 minor:411 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6cd82820d0547eaa5d568e5aa00374e422051909ee4a4a338f7992765c27855/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6cd82820d0547eaa5d568e5aa00374e422051909ee4a4a338f7992765c27855/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7a612a206c05dbbc84b2330e873b6f105a49a7aa4c7cbb3de9fda66e9e124e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7a612a206c05dbbc84b2330e873b6f105a49a7aa4c7cbb3de9fda66e9e124e6/userdata/shm major:0 minor:413 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fbe58304c85541f8664e66f2d8e258a24f94270321951fbbc1e91fb4125b3b83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fbe58304c85541f8664e66f2d8e258a24f94270321951fbbc1e91fb4125b3b83/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0001caf2-fffa-457e-8ab5-9574e11aced5/volumes/kubernetes.io~projected/kube-api-access-wgbs9:{mountpoint:/var/lib/kubelet/pods/0001caf2-fffa-457e-8ab5-9574e11aced5/volumes/kubernetes.io~projected/kube-api-access-wgbs9 major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ddcb000-6271-4c42-a911-b3a4d14711c3/volumes/kubernetes.io~projected/kube-api-access-5h2ng:{mountpoint:/var/lib/kubelet/pods/0ddcb000-6271-4c42-a911-b3a4d14711c3/volumes/kubernetes.io~projected/kube-api-access-5h2ng major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volume-subpaths/web-config/config-reloader/2 major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volume-subpaths/web-config/prometheus/15 major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~empty-dir/config-out major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~projected/kube-api-access-f7g4d:{mountpoint:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~projected/kube-api-access-f7g4d major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~projected/tls-assets major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/config major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:433 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/web-config major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10982725-01b4-4d8e-8ad4-44d43608e6b0/volumes/kubernetes.io~projected/kube-api-access-2zkkn:{mountpoint:/var/lib/kubelet/pods/10982725-01b4-4d8e-8ad4-44d43608e6b0/volumes/kubernetes.io~projected/kube-api-access-2zkkn major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19997935-7a8e-4747-bb75-4532f1a72db5/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/19997935-7a8e-4747-bb75-4532f1a72db5/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19997935-7a8e-4747-bb75-4532f1a72db5/volumes/kubernetes.io~projected/kube-api-access-g72wk:{mountpoint:/var/lib/kubelet/pods/19997935-7a8e-4747-bb75-4532f1a72db5/volumes/kubernetes.io~projected/kube-api-access-g72wk major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19997935-7a8e-4747-bb75-4532f1a72db5/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/19997935-7a8e-4747-bb75-4532f1a72db5/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d508cdb-d418-4f6c-89c5-45e02e93f9d0/volumes/kubernetes.io~projected/kube-api-access-jw58z:{mountpoint:/var/lib/kubelet/pods/1d508cdb-d418-4f6c-89c5-45e02e93f9d0/volumes/kubernetes.io~projected/kube-api-access-jw58z major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d508cdb-d418-4f6c-89c5-45e02e93f9d0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1d508cdb-d418-4f6c-89c5-45e02e93f9d0/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b1f827c-2d6d-4d1d-ac35-23324ff897fa/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/3b1f827c-2d6d-4d1d-ac35-23324ff897fa/volumes/kubernetes.io~secret/tls-certificates major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eb20209-5da1-4ef7-8625-b06c2b6493ef/volumes/kubernetes.io~projected/kube-api-access-xzzc8:{mountpoint:/var/lib/kubelet/pods/3eb20209-5da1-4ef7-8625-b06c2b6493ef/volumes/kubernetes.io~projected/kube-api-access-xzzc8 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eb20209-5da1-4ef7-8625-b06c2b6493ef/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3eb20209-5da1-4ef7-8625-b06c2b6493ef/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eb20209-5da1-4ef7-8625-b06c2b6493ef/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/3eb20209-5da1-4ef7-8625-b06c2b6493ef/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/420b750b-2a29-4432-84f7-0d7709493a41/volumes/kubernetes.io~projected/kube-api-access-4nn8g:{mountpoint:/var/lib/kubelet/pods/420b750b-2a29-4432-84f7-0d7709493a41/volumes/kubernetes.io~projected/kube-api-access-4nn8g major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/420b750b-2a29-4432-84f7-0d7709493a41/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/420b750b-2a29-4432-84f7-0d7709493a41/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/420b750b-2a29-4432-84f7-0d7709493a41/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/420b750b-2a29-4432-84f7-0d7709493a41/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44293dda-24ae-4aae-aa97-1e262c3186de/volumes/kubernetes.io~projected/kube-api-access-w4fnf:{mountpoint:/var/lib/kubelet/pods/44293dda-24ae-4aae-aa97-1e262c3186de/volumes/kubernetes.io~projected/kube-api-access-w4fnf major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44293dda-24ae-4aae-aa97-1e262c3186de/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/44293dda-24ae-4aae-aa97-1e262c3186de/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44293dda-24ae-4aae-aa97-1e262c3186de/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/44293dda-24ae-4aae-aa97-1e262c3186de/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46932d2f-65ad-4ac4-b422-f4991d35cbd2/volumes/kubernetes.io~projected/kube-api-access-x4p2b:{mountpoint:/var/lib/kubelet/pods/46932d2f-65ad-4ac4-b422-f4991d35cbd2/volumes/kubernetes.io~projected/kube-api-access-x4p2b major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fe6a899-6382-488a-b52d-e20c68445e67/volumes/kubernetes.io~projected/kube-api-access-9wz4p:{mountpoint:/var/lib/kubelet/pods/4fe6a899-6382-488a-b52d-e20c68445e67/volumes/kubernetes.io~projected/kube-api-access-9wz4p major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/590e650a-39b7-4696-8bf9-c105a87fb012/volumes/kubernetes.io~projected/kube-api-access-w9whs:{mountpoint:/var/lib/kubelet/pods/590e650a-39b7-4696-8bf9-c105a87fb012/volumes/kubernetes.io~projected/kube-api-access-w9whs major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ea8cfb3-a932-4689-8eda-f25dc44ca94a/volumes/kubernetes.io~projected/kube-api-access-klngr:{mountpoint:/var/lib/kubelet/pods/5ea8cfb3-a932-4689-8eda-f25dc44ca94a/volumes/kubernetes.io~projected/kube-api-access-klngr major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ea8cfb3-a932-4689-8eda-f25dc44ca94a/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/5ea8cfb3-a932-4689-8eda-f25dc44ca94a/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ea8cfb3-a932-4689-8eda-f25dc44ca94a/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/5ea8cfb3-a932-4689-8eda-f25dc44ca94a/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a305968-1cca-480f-9360-aeb99c0ef2de/volumes/kubernetes.io~projected/kube-api-access-pdrp4:{mountpoint:/var/lib/kubelet/pods/6a305968-1cca-480f-9360-aeb99c0ef2de/volumes/kubernetes.io~projected/kube-api-access-pdrp4 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8411ba88-6e16-48e1-ab47-a33b4d5f0766/volumes/kubernetes.io~projected/kube-api-access-6cf2w:{mountpoint:/var/lib/kubelet/pods/8411ba88-6e16-48e1-ab47-a33b4d5f0766/volumes/kubernetes.io~projected/kube-api-access-6cf2w major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8411ba88-6e16-48e1-ab47-a33b4d5f0766/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/8411ba88-6e16-48e1-ab47-a33b4d5f0766/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95f11ca1-5d71-4135-bf16-0fe6c24706c0/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/95f11ca1-5d71-4135-bf16-0fe6c24706c0/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:405 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a459d678-44f2-4eed-9cb2-f370f1dcc8c1/volumes/kubernetes.io~projected/kube-api-access-xc7c8:{mountpoint:/var/lib/kubelet/pods/a459d678-44f2-4eed-9cb2-f370f1dcc8c1/volumes/kubernetes.io~projected/kube-api-access-xc7c8 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a459d678-44f2-4eed-9cb2-f370f1dcc8c1/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/a459d678-44f2-4eed-9cb2-f370f1dcc8c1/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a459d678-44f2-4eed-9cb2-f370f1dcc8c1/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/a459d678-44f2-4eed-9cb2-f370f1dcc8c1/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7a37dcb-96d5-4c31-a7d7-08f663036a86/volumes/kubernetes.io~projected/kube-api-access-877qg:{mountpoint:/var/lib/kubelet/pods/d7a37dcb-96d5-4c31-a7d7-08f663036a86/volumes/kubernetes.io~projected/kube-api-access-877qg major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7a37dcb-96d5-4c31-a7d7-08f663036a86/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/d7a37dcb-96d5-4c31-a7d7-08f663036a86/volumes/kubernetes.io~secret/metrics-certs major:0 minor:409 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dff6e046-945d-4553-ba94-908dbb1a0c23/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/dff6e046-945d-4553-ba94-908dbb1a0c23/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0407316-563c-489c-9826-3c1372a6fbff/volumes/kubernetes.io~projected/kube-api-access-9fkx6:{mountpoint:/var/lib/kubelet/pods/e0407316-563c-489c-9826-3c1372a6fbff/volumes/kubernetes.io~projected/kube-api-access-9fkx6 major:0 minor:458 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0407316-563c-489c-9826-3c1372a6fbff/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/e0407316-563c-489c-9826-3c1372a6fbff/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:452 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0407316-563c-489c-9826-3c1372a6fbff/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/e0407316-563c-489c-9826-3c1372a6fbff/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:457 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0ac6e99-4947-4703-a31d-091a6cc9b9d7/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e0ac6e99-4947-4703-a31d-091a6cc9b9d7/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0ac6e99-4947-4703-a31d-091a6cc9b9d7/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e0ac6e99-4947-4703-a31d-091a6cc9b9d7/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0ac6e99-4947-4703-a31d-091a6cc9b9d7/volumes/kubernetes.io~projected/kube-api-access-w4qm4:{mountpoint:/var/lib/kubelet/pods/e0ac6e99-4947-4703-a31d-091a6cc9b9d7/volumes/kubernetes.io~projected/kube-api-access-w4qm4 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ece532ab-396d-46f7-9de2-9e19563a5528/volumes/kubernetes.io~projected/kube-api-access-pdc49:{mountpoint:/var/lib/kubelet/pods/ece532ab-396d-46f7-9de2-9e19563a5528/volumes/kubernetes.io~projected/kube-api-access-pdc49 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ece532ab-396d-46f7-9de2-9e19563a5528/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/ece532ab-396d-46f7-9de2-9e19563a5528/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f019c85d-f532-4848-905b-b7e4480adba1/volumes/kubernetes.io~projected/kube-api-access-89kxn:{mountpoint:/var/lib/kubelet/pods/f019c85d-f532-4848-905b-b7e4480adba1/volumes/kubernetes.io~projected/kube-api-access-89kxn major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f019c85d-f532-4848-905b-b7e4480adba1/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f019c85d-f532-4848-905b-b7e4480adba1/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f019c85d-f532-4848-905b-b7e4480adba1/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/f019c85d-f532-4848-905b-b7e4480adba1/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volume-subpaths/web-config/alertmanager/9 major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volume-subpaths/web-config/config-reloader/6 major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~empty-dir/config-out major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~projected/kube-api-access-hd89k:{mountpoint:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~projected/kube-api-access-hd89k major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~projected/tls-assets major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/config-volume major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/web-config major:0 minor:334 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/bada1415e28f375e9fd956e7ef0f95f2d927ed6e2fdc5dc3f23c785689c6a4c9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/3cf43e6c3b07f71a14f1a656ddc6b6beba86b458cb0b99cf7929a0d066c5c50a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/2dfb2c3f2712a32e01989a1cf9019b97e8fc400de2b0b751fbdde21220fddf29/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-107:{mountpoint:/var/lib/containers/storage/overlay/d9887903aa39cefdb73031dd3f7b75caa75fe0e40b2b40b82afb3debe159ca2a/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a5e66d8a81b459122cff40875287127ee79eb101b6ebfa3735803767c06f89f5/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/371158c6c97886688315f85554ab295447690f4a79cd0f1e5b04f5a0369ec0f0/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/cd46659e69d20e08c19b822c40cb829f6b1637d5b854f32bb0554e3ff7efc50d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b44c962ff2ac3e35c1d728c7599014fc972f4b23677ae7ab06cfeb6b249f57cf/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d3b4cfebeb9cfe65c8b023553092e5cce4b9dbaa8f4e26b59c1be66af38512bb/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/7dfbb602c62e77496ff11c73400215f55bed65f769dc58e46e1347190330d461/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/0ccb30b38efe9f54fe3a211760411cd7e73a7772056be69202f54db6caab2465/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/a0331432658e48ae70caddcb33689bfd178cf8be7315963faa109fbadeceed03/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a035cf36b41018697007b109b1fceda6abdfac26ab6a58e9c8bcd964b76c2ca6/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/71fa6b8e7a32bc446bb719b88b2142e129aa8e203647ecc98f6a085b26bfc8eb/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6af94e786ab715a9505204fb5bdea66950b31c43d3770948b2fdc27cc7fbf783/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/29cdb4ea839cfd1ba65ede01dde034146f674e0f9f1624e082eacabc8717e1e1/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8851846f317edf4b06676f9d96209b822216cb62b22586036c262dc256f52900/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d5a61157717b9669fc1be7af0731f4503449ca0988675f7af797e6974bc1c6d0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/082f54d159f20afc53c12cbce3e631f7af6990c034d3c04526fe97bfa97ce318/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/4437aba67d605155e4c52c3d27dba195f9ebd0092c25ddaf079650a5864d7c3b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ddbe7686de37722090598672f17174d8ef78534e7537b2e864ac4cf1a3aebbfa/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/b7954be62547ded5496868f85cab88ab3c699f552ace55b36f1d80c053888dee/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/e25cdce204191bd876e129856a0143513d4043a7694e16a12dd9c794baa70744/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/5f74c1300d31c0ffaa4fec7ca4e0a566eb33f94bb56547811372169e418469d8/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/7f312fffdc417a6b4f96ae4e638e5e4e19f5443d750bfb569b8628d44c4559a5/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/ffd6147fa8e410369bd5fb64c7d572044876cc719dc5328384f65358c136654c/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/ef782ad7e767bc75aa1706c8f68b0616f78328b4b757e93a7eb45fcfabd173e0/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/dc79bcc1a840a6407bd5b5093ec2a5e1475f9dbcefdf47b1d6a367f62508bd39/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/3c5ecbac25a33f635e1043aa18ba23e6c57438f07c2290481cdf9d891f862d5f/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/08072b5b1abdd8ed4f0e4a123721a38b4af92354cf055858c530a179b779f77f/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/e3912e952259c9405c92d24cddca57cf7e12fed6a4509779850b83d9dcba9b42/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/767d99a1a2f2d2cb4bcbb968c54ac1c7cd6a2e5992af66a78c5827f165239ffd/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/05b8f891189d63ef03563bc1afb5055db424e5c396ac328f0e5aea1ba770c20b/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/5c2b10f85266734d41104391d72dccd3bd1dde7d7b61e24d03ab591ade1eea8a/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/b3d9d6842893289df14bf817e0f426395ab76c9892f7e31f0502a2d10fbe29c7/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/831365effd068ef77b67652e89b2f853e7dea17e2f42e888860de8874868558f/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/b540239fd69fffdbaa01a2d57c3679264512411be008ecf48aef4169de549412/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/7921521779a8c199e96a9c89881e5c7e9c948d201fc3c11c104954e503aabaca/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/6422048573b34e10e8b8487f10acbee098c88d51035dad4069c9b82f389e29ae/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/b4f4e80dc25c7e26846c65a4b6e8c99147ce30f43be3e88896c51456025165f4/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/2ac7b86d63f7cf020a684b144955b54917e27a446f5cde4be8f26be7c4f9784e/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/02d86451c884d6e1a50ed1e9b465012902d5cb81b1b3facb56bebbfba25a6669/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/e893b7838fffad269ec9b4a2fb02c6168373c0ee9c67b89a3899645b3b0e0490/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/6d69b933cea9879b2c3f054bd3e694f014582338a3006d46817d5d250c084942/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/ae31c881a4a53ff225c86d034d5d130760c0881b67235230814957d6e166ea00/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/09ac3af6e3bca1ac6b14c55aefbf1896a595757e577d780f639195e74081578e/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/e6b7a399672dfc27614f413c7008f3d03125395df8cdf4918d680e96c9959c49/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/e945584d4080a37b8d80a7d2d9641314beb31799474fcbdb595c551d0c886ad8/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/c3e5aef6e68e9c5fa25f2f245643d3f0046dad9269d1c85faad8169065c5a037/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/afd17751b3af45bb2023d59061aff6ecf48e318b55ba1b26c8be3b5844c6f2a0/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/30f90d2f1612f6d66c856e2ea5b88384ea13c44857cc9b22f70c1f65fa9d0286/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/436744bd17e4d39724a04d6fde91c92e6b9fce629cdee9337fdf796fcbf529ed/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/96ad10cd624a7530606a77c30ab2dd88f1b1b0eff44f766c1e26b77b0197e912/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/e99b66f995cc7bf06f3433d64b69cfe787bf1206dae2e6b01339abdbdd773610/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/6a2fed5418637de4f18052f2e6d76aaa3de68a4013e4b2ffaad3096f7c79c374/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/a4c99f53b0893846c20b5b2017770f0d4f5613e3ce331e57eb43c17cf7dc9052/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/c1067806a51722b72499a25b4853a6510a77604c5ad2451122c64cae627b6ae5/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/57e320b4b4b62ddaeef83f415820ff82485f6e52d55717d65e6cb4c584d06476/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/3e5fb78a7e4d3d7c2d0003a1620f359f4659c90e1b023fcd2b70a0a448403fdf/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/ef66b2d076c03d1fa7c4ec7cf747fd4476240b4759461a35abcfaf36464ab95c/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-439:{mountpoint:/var/lib/containers/storage/overlay/f49665fe93785268ea413821f23247e48c5930bc9ceec6d9026397cf2ce835e1/merged major:0 minor:439 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/3e8f1d21e407eda05588b33a3945df222df84170c9b1ed9413dcd05d3d709556/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/dad4ea051fa846d594de8b15f7aa4353cde6f6df4ddeaa201623d9f018aebd76/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-459:{mountpoint:/var/lib/containers/storage/overlay/9a28ca3fe5cd62470b60352955f12d432fda1c1b3bff93891441cc2b1d10fc12/merged major:0 minor:459 fsType:overlay blockSize:0} overlay_0-461:{mountpoint:/var/lib/containers/storage/overlay/c1fa7f9fe045bc0665af79b88596553ee15bbdeabe50380f54685f60639850e6/merged major:0 minor:461 fsType:overlay blockSize:0} overlay_0-472:{mountpoint:/var/lib/containers/storage/overlay/9c4ac8dbc24bc14f56de64f04289a2ac453ab80019f99df2615ab8c5114d1004/merged major:0 minor:472 fsType:overlay blockSize:0} overlay_0-474:{mountpoint:/var/lib/containers/storage/overlay/2877a376593ab72def9db5b303869dabd1b60edd42c80b60ba6269190e672d18/merged major:0 minor:474 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/56ca26f837d23d3d80a4362f2afb043ffd9414c6433d68122424ef58252bd7ee/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/6b2ffe3321d191969a327e08b9dbb99b153cb05e902f8d6134b7a0a669234577/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/441d56cad34da2965ba27227eb8ef9651093b7193e10c99b6d99b101c2b91d1f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/dbbefc906b8e02d0974c45c9fda42dff809640602c2b766c086b70b2a41b6ddb/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/8c57162bf57ac96d5a91306aea18b91ee8d103cfe2b7fb60e96b7efb4fceb348/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/3020bfa357efc52bec4045c5f63c8a3c2ba29ea1cd96090f930588c2480ca6fe/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/66cf135b6e9f14e7df027f8dbaec6d8205e0ab3bb8e3a1465f2ec4313be05297/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/f1a08e3c8d4cba35a9733b19c9a2af3755e6e72a3b7c3cba80fec99d8787a172/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/43937fce89575506a14c840e6b00e38ac966ff261fdcf9afcb96b9f8e20d9d20/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e8c8b351db912279f8ebc8516cbf6e2d5039990171a8dd1d40c2e20dd91baa1e/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e28028ca2291bb0d98674e1a7695b802cafbb1e905819ca647b7cbdd22c2e228/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 20:27:17.141991 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.141345 5755 manager.go:217] Machine: {Timestamp:2026-10-30 20:27:17.14085787 +0000 UTC m=+0.067414777 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100435 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22766d10a560c37c482443d2839b55 SystemUUID:ec22766d-10a5-60c3-7c48-2443d2839b55 BootID:acb9433a-3f30-4886-851b-4e9b11c2b1fe Filesystems:[{Device:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:428 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/19997935-7a8e-4747-bb75-4532f1a72db5/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~projected/kube-api-access-hd89k DeviceMajor:0 DeviceMinor:345 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ece532ab-396d-46f7-9de2-9e19563a5528/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a459d678-44f2-4eed-9cb2-f370f1dcc8c1/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0ac6e99-4947-4703-a31d-091a6cc9b9d7/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4eff3f83181a243c24aabb7a7daeac647728e7bdfb76044e570a31316e826e1b/userdata/shm DeviceMajor:0 DeviceMinor:277 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:336 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:430 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/947e91e49a01c04511694a679764c49788f373f6bf7e637231fd11bc0f5f862b/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d508cdb-d418-4f6c-89c5-45e02e93f9d0/volumes/kubernetes.io~projected/kube-api-access-jw58z DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-459 DeviceMajor:0 DeviceMinor:459 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0ac6e99-4947-4703-a31d-091a6cc9b9d7/volumes/kubernetes.io~projected/kube-api-access-w4qm4 DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3eb20209-5da1-4ef7-8625-b06c2b6493ef/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/420b750b-2a29-4432-84f7-0d7709493a41/volumes/kubernetes.io~projected/kube-api-access-4nn8g DeviceMajor:0 DeviceMinor:299 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/164bce9e38bd01c82a5295a3639118dbe978703f5028948ab9f98eb52616cb77/userdata/shm DeviceMajor:0 DeviceMinor:367 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85a7e16f66ac99d6a3befab3fb2fe7afe59b9000a12c79c93cd416f45e23518e/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:333 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/44293dda-24ae-4aae-aa97-1e262c3186de/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:365 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:333 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7a37dcb-96d5-4c31-a7d7-08f663036a86/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:409 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5002587a4732072cddd53b1bf39d63e1ad3aaf071fc57b426faf8c77402ec03/userdata/shm DeviceMajor:0 DeviceMinor:351 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0001caf2-fffa-457e-8ab5-9574e11aced5/volumes/kubernetes.io~projected/kube-api-access-wgbs9 DeviceMajor:0 DeviceMinor:410 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea970a988820430430f16480ff40c16cc859236a183338f277d48931fe0302f3/userdata/shm DeviceMajor:0 DeviceMinor:411 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/0ddcb000-6271-4c42-a911-b3a4d14711c3/volumes/kubernetes.io~projected/kube-api-access-5h2ng DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4dbfa8cf88c5a111ad23499c0d035296b7b8e9a30edd98d6e13536a6b620899b/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:426 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:431 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e0407316-563c-489c-9826-3c1372a6fbff/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:452 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a459d678-44f2-4eed-9cb2-f370f1dcc8c1/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46932d2f-65ad-4ac4-b422-f4991d35cbd2/volumes/kubernetes.io~projected/kube-api-access-x4p2b DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f019c85d-f532-4848-905b-b7e4480adba1/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:290 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7a612a206c05dbbc84b2330e873b6f105a49a7aa4c7cbb3de9fda66e9e124e6/userdata/shm DeviceMajor:0 DeviceMinor:413 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4fe6a899-6382-488a-b52d-e20c68445e67/volumes/kubernetes.io~projected/kube-api-access-9wz4p DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ece532ab-396d-46f7-9de2-9e19563a5528/volumes/kubernetes.io~projected/kube-api-access-pdc49 DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3eb20209-5da1-4ef7-8625-b06c2b6493ef/volumes/kubernetes.io~projected/kube-api-access-xzzc8 DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:432 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e0ac6e99-4947-4703-a31d-091a6cc9b9d7/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-107 DeviceMajor:0 DeviceMinor:107 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8411ba88-6e16-48e1-ab47-a33b4d5f0766/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a459d678-44f2-4eed-9cb2-f370f1dcc8c1/volumes/kubernetes.io~projected/kube-api-access-xc7c8 DeviceMajor:0 DeviceMinor:235 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/420b750b-2a29-4432-84f7-0d7709493a41/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:295 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/420b750b-2a29-4432-84f7-0d7709493a41/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:296 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dff6e046-945d-4553-ba94-908dbb1a0c23/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5fd458ac66e32419632ae0455910180d935049c4b6fdbd6134b35228130793f2/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ea8cfb3-a932-4689-8eda-f25dc44ca94a/volumes/kubernetes.io~projected/kube-api-access-klngr DeviceMajor:0 DeviceMinor:276 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f019c85d-f532-4848-905b-b7e4480adba1/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:291 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d7a37dcb-96d5-4c31-a7d7-08f663036a86/volumes/kubernetes.io~projected/kube-api-access-877qg DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13f03e50512a578a5ec3e178a77dcb47dc4637aa5fe2f0fe074b2e67172dba83/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5ea8cfb3-a932-4689-8eda-f25dc44ca94a/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:275 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/792ee3a0c0bf66cf265bc5e3ce143c98ada063b623d998319e6c6d73ce70a5ec/userdata/shm DeviceMajor:0 DeviceMinor:297 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:426 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-439 DeviceMajor:0 DeviceMinor:439 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:340 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e0407316-563c-489c-9826-3c1372a6fbff/volumes/kubernetes.io~projected/kube-api-access-9fkx6 DeviceMajor:0 DeviceMinor:458 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:426 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/10982725-01b4-4d8e-8ad4-44d43608e6b0/volumes/kubernetes.io~projected/kube-api-access-2zkkn DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fbe58304c85541f8664e66f2d8e258a24f94270321951fbbc1e91fb4125b3b83/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/44293dda-24ae-4aae-aa97-1e262c3186de/volumes/kubernetes.io~projected/kube-api-access-w4fnf DeviceMajor:0 DeviceMinor:366 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:426 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/590e650a-39b7-4696-8bf9-c105a87fb012/volumes/kubernetes.io~projected/kube-api-access-w9whs DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:429 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81bd1aa7a4ea1b9b050060623a14ecd686bad9ce306fb444336a134c6c1c4b9d/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45bdf7a357596324cf935426759a332c7c9d969bc2a2c8ba7a682663f0c26967/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:425 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/365cb554216b8fa9574c9b4c326fc8927b9d610d05f3477e9fa04a4462aaf968/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19997935-7a8e-4747-bb75-4532f1a72db5/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3b1f827c-2d6d-4d1d-ac35-23324ff897fa/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:338 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d508cdb-d418-4f6c-89c5-45e02e93f9d0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e78b32209ac052ac75dd366a4bfdbe197ee2de2c77c81bbc38acd54b4b9b3a75/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5ea8cfb3-a932-4689-8eda-f25dc44ca94a/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:270 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c7aa12704a9e14c288dd5a706e5ef45b8fe03363814b5a5f135fe87662138dd/userdata/shm DeviceMajor:0 DeviceMinor:471 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9570b49e052d07f7a47404631ce0b9342499670ec3e6d0f0c0f0efa117106b70/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d72a74997b080c5ebfbe201aadb40f621cb3bd41f963e38b68ae2c3cdea79f84/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:424 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1b511941563444d88c34017d11fca27ce03fbfd82d708b2aff9eeb6c6c2f534/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ed2b64f18b5b5b9da7c1c8d45f3c754b03835c8b12014432f6cb0ec361daf62/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:337 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:423 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e618f27663d49d4bd8025e8fee1cc556fa1117ab56538d3db26cb14e57581d1/userdata/shm DeviceMajor:0 DeviceMinor:435 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/19997935-7a8e-4747-bb75-4532f1a72db5/volumes/kubernetes.io~projected/kube-api-access-g72wk DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/879c830c59794efe325ec0e1c7671029cb1e7e5fdfd7af2736ab352f6eab8cba/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8411ba88-6e16-48e1-ab47-a33b4d5f0766/volumes/kubernetes.io~projected/kube-api-access-6cf2w DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:433 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5bbfc5789732b84c826edfc49bf665c799b1cb2ee44397c9ff2eb77998daba5/userdata/shm DeviceMajor:0 DeviceMinor:415 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:427 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-461 DeviceMajor:0 DeviceMinor:461 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-472 DeviceMajor:0 DeviceMinor:472 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:331 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/44293dda-24ae-4aae-aa97-1e262c3186de/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:361 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes/kubernetes.io~projected/kube-api-access-f7g4d DeviceMajor:0 DeviceMinor:434 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-474 DeviceMajor:0 DeviceMinor:474 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a305968-1cca-480f-9360-aeb99c0ef2de/volumes/kubernetes.io~projected/kube-api-access-pdrp4 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b9ee59e00f9f10151f45bc9eeda5290f733fa47f3e1c8e0aa5fde6fe999f48c/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3eb20209-5da1-4ef7-8625-b06c2b6493ef/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95f11ca1-5d71-4135-bf16-0fe6c24706c0/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:405 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e0407316-563c-489c-9826-3c1372a6fbff/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:457 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6cd82820d0547eaa5d568e5aa00374e422051909ee4a4a338f7992765c27855/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f019c85d-f532-4848-905b-b7e4480adba1/volumes/kubernetes.io~projected/kube-api-access-89kxn DeviceMajor:0 DeviceMinor:294 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:164bce9e38bd01c MacAddress:ca:64:ea:21:d1:dc Speed:10000 Mtu:8901} {Name:45bdf7a35759632 MacAddress:fe:aa:b1:e5:a0:9c Speed:10000 Mtu:8901} {Name:4eff3f83181a243 MacAddress:26:e2:57:cf:f4:14 Speed:10000 Mtu:8901} {Name:6e618f27663d49d MacAddress:fa:0f:e9:ad:d6:01 Speed:10000 Mtu:8901} {Name:792ee3a0c0bf66c MacAddress:ba:b7:15:05:33:51 Speed:10000 Mtu:8901} {Name:85a7e16f66ac99d MacAddress:aa:07:a0:80:e8:10 Speed:10000 Mtu:8901} {Name:9570b49e052d07f MacAddress:b6:27:58:0a:75:3e Speed:10000 Mtu:8901} {Name:9c7aa12704a9e14 MacAddress:d2:46:cd:48:54:93 Speed:10000 Mtu:8901} {Name:a5002587a473207 MacAddress:1e:b5:8e:68:26:28 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:66:2a:fa:a6:df Speed:0 Mtu:9001} {Name:br-int MacAddress:6a:77:4f:53:db:5e Speed:0 Mtu:8901} {Name:c5bbfc5789732b8 MacAddress:0e:30:3f:20:a0:54 Speed:10000 Mtu:8901} {Name:d72a74997b080c5 MacAddress:4e:6f:dc:4a:48:ac Speed:10000 Mtu:8901} {Name:ea970a988820430 MacAddress:fe:f1:65:90:2c:33 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:66:2a:fa:a6:df Speed:0 Mtu:9001} {Name:f6cd82820d0547e MacAddress:72:f7:55:3a:d0:e4 Speed:10000 Mtu:8901} {Name:f7a612a206c05db MacAddress:fa:66:dd:49:c7:4f Speed:10000 Mtu:8901} {Name:fbe58304c85541f MacAddress:92:e8:39:5b:d3:8f Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:aa:54:6d:5f:fc:0f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:aa:3a:48:4f:4d:9d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 20:27:17.142236 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.141999 5755 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 20:27:17.142236 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.142059 5755 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 20:27:17.142394 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.142356 5755 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 20:27:17.142551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.142411 5755 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-195.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 20:27:17.142598 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.142554 5755 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 20:27:17.142598 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.142562 5755 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 20:27:17.142598 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.142581 5755 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:27:17.142672 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.142602 5755 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:27:17.142718 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.142710 5755 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 20:27:17.142885 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.142875 5755 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 20:27:17.143006 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.142997 5755 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 20:27:17.143047 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.143038 5755 kubelet.go:493] "Attempting to sync node with API server" Oct 30 20:27:17.143092 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.143049 5755 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 20:27:17.143092 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.143060 5755 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 20:27:17.143092 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.143069 5755 kubelet.go:405] "Adding apiserver pod source" Oct 30 20:27:17.143092 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.143078 5755 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 20:27:17.143958 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.143940 5755 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 20:27:17.143958 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.143958 5755 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:27:17.144587 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.144566 5755 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 20:27:17.156846 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.156827 5755 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 20:27:17.156846 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.156844 5755 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:27:17.157019 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.156992 5755 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 20:27:17.157255 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.157238 5755 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 20:27:17.157320 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.157279 5755 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 20:27:17.157474 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.157460 5755 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 20:27:17.157546 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.157482 5755 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 20:27:17.157546 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.157493 5755 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 20:27:17.157546 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.157502 5755 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 20:27:17.157546 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.157512 5755 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 20:27:17.157546 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.157521 5755 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 20:27:17.157546 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.157530 5755 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 20:27:17.157546 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.157539 5755 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 20:27:17.157838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.157549 5755 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 20:27:17.157838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.157558 5755 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 20:27:17.157838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.157579 5755 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 20:27:17.157838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.157594 5755 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 20:27:17.157838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.157637 5755 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 20:27:17.157838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.157646 5755 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 20:27:17.164156 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.164129 5755 server.go:1263] "Started kubelet" Oct 30 20:27:17.164315 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.164256 5755 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 20:27:17.164411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.164345 5755 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 20:27:17.164669 ip-10-0-139-195 systemd[1]: Started Kubernetes Kubelet. Oct 30 20:27:17.164824 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.164724 5755 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 20:27:17.165976 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.165957 5755 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 20:27:17.166088 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.166077 5755 server.go:317] "Adding debug handlers to kubelet server" Oct 30 20:27:17.168116 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.168098 5755 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 20:27:17.168229 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.168211 5755 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 20:27:17.168229 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:17.168201 5755 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-195.us-west-1.compute.internal\" not found" Oct 30 20:27:17.168336 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.168260 5755 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 20:27:17.168336 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.168267 5755 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 20:27:17.170157 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.168513 5755 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:21:06 +0000 UTC" deadline="2028-06-04 06:25:11.905173781 +0000 UTC" Oct 30 20:27:17.170157 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.168539 5755 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13977h57m54.736638461s" Oct 30 20:27:17.170157 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.168557 5755 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 20:27:17.173267 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.173245 5755 factory.go:55] Registering systemd factory Oct 30 20:27:17.173267 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.173267 5755 factory.go:223] Registration of the systemd container factory successfully Oct 30 20:27:17.173595 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.173579 5755 factory.go:153] Registering CRI-O factory Oct 30 20:27:17.173740 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.173599 5755 factory.go:223] Registration of the crio container factory successfully Oct 30 20:27:17.173740 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.173658 5755 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 20:27:17.173740 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.173679 5755 factory.go:103] Registering Raw factory Oct 30 20:27:17.173740 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.173696 5755 manager.go:1199] Started watching for new ooms in manager Oct 30 20:27:17.174072 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.174059 5755 manager.go:319] Starting recovery of all containers Oct 30 20:27:17.178994 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.178943 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 20:27:17.179095 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179002 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a459d678-44f2-4eed-9cb2-f370f1dcc8c1" volumeName="kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-service-ca" seLinuxMountContext="" Oct 30 20:27:17.179095 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179024 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0407316-563c-489c-9826-3c1372a6fbff" volumeName="kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-oauth-config" seLinuxMountContext="" Oct 30 20:27:17.179095 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179042 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b04324-e2ac-435b-8756-3812700e9af7" volumeName="kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 20:27:17.179095 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179060 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19997935-7a8e-4747-bb75-4532f1a72db5" volumeName="kubernetes.io/secret/19997935-7a8e-4747-bb75-4532f1a72db5-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 20:27:17.179095 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179076 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44293dda-24ae-4aae-aa97-1e262c3186de" volumeName="kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-serving-cert" seLinuxMountContext="" Oct 30 20:27:17.179095 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179093 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f019c85d-f532-4848-905b-b7e4480adba1" volumeName="kubernetes.io/configmap/f019c85d-f532-4848-905b-b7e4480adba1-metrics-client-ca" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179110 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f019c85d-f532-4848-905b-b7e4480adba1" volumeName="kubernetes.io/configmap/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179132 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ddcb000-6271-4c42-a911-b3a4d14711c3" volumeName="kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-cni-binary-copy" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179149 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="420b750b-2a29-4432-84f7-0d7709493a41" volumeName="kubernetes.io/projected/420b750b-2a29-4432-84f7-0d7709493a41-kube-api-access-4nn8g" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179164 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="420b750b-2a29-4432-84f7-0d7709493a41" volumeName="kubernetes.io/secret/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-tls" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179181 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ece532ab-396d-46f7-9de2-9e19563a5528" volumeName="kubernetes.io/configmap/ece532ab-396d-46f7-9de2-9e19563a5528-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179195 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="420b750b-2a29-4432-84f7-0d7709493a41" volumeName="kubernetes.io/empty-dir/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-textfile" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179209 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44293dda-24ae-4aae-aa97-1e262c3186de" volumeName="kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-console-config" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179233 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179251 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46932d2f-65ad-4ac4-b422-f4991d35cbd2" volumeName="kubernetes.io/configmap/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-daemon-config" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179266 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95f11ca1-5d71-4135-bf16-0fe6c24706c0" volumeName="kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179282 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179296 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dff6e046-945d-4553-ba94-908dbb1a0c23" volumeName="kubernetes.io/secret/dff6e046-945d-4553-ba94-908dbb1a0c23-agent-certs" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179308 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44293dda-24ae-4aae-aa97-1e262c3186de" volumeName="kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-oauth-serving-cert" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179323 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fe6a899-6382-488a-b52d-e20c68445e67" volumeName="kubernetes.io/projected/4fe6a899-6382-488a-b52d-e20c68445e67-kube-api-access-9wz4p" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179331 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="590e650a-39b7-4696-8bf9-c105a87fb012" volumeName="kubernetes.io/projected/590e650a-39b7-4696-8bf9-c105a87fb012-kube-api-access-w9whs" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179342 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea8cfb3-a932-4689-8eda-f25dc44ca94a" volumeName="kubernetes.io/empty-dir/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-tmp" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179349 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8411ba88-6e16-48e1-ab47-a33b4d5f0766" volumeName="kubernetes.io/secret/8411ba88-6e16-48e1-ab47-a33b4d5f0766-metrics-tls" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179358 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a459d678-44f2-4eed-9cb2-f370f1dcc8c1" volumeName="kubernetes.io/projected/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-kube-api-access-xc7c8" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179368 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7a37dcb-96d5-4c31-a7d7-08f663036a86" volumeName="kubernetes.io/projected/d7a37dcb-96d5-4c31-a7d7-08f663036a86-kube-api-access-877qg" seLinuxMountContext="" Oct 30 20:27:17.179405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179401 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0407316-563c-489c-9826-3c1372a6fbff" volumeName="kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-console-config" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179420 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ddcb000-6271-4c42-a911-b3a4d14711c3" volumeName="kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179438 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ddcb000-6271-4c42-a911-b3a4d14711c3" volumeName="kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179457 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179482 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d508cdb-d418-4f6c-89c5-45e02e93f9d0" volumeName="kubernetes.io/secret/1d508cdb-d418-4f6c-89c5-45e02e93f9d0-cert" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179499 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a459d678-44f2-4eed-9cb2-f370f1dcc8c1" volumeName="kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-config" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179514 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f019c85d-f532-4848-905b-b7e4480adba1" volumeName="kubernetes.io/projected/f019c85d-f532-4848-905b-b7e4480adba1-kube-api-access-89kxn" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179529 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b04324-e2ac-435b-8756-3812700e9af7" volumeName="kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179546 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a305968-1cca-480f-9360-aeb99c0ef2de" volumeName="kubernetes.io/projected/6a305968-1cca-480f-9360-aeb99c0ef2de-kube-api-access-pdrp4" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179559 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8411ba88-6e16-48e1-ab47-a33b4d5f0766" volumeName="kubernetes.io/empty-dir/8411ba88-6e16-48e1-ab47-a33b4d5f0766-tmp-dir" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179575 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f019c85d-f532-4848-905b-b7e4480adba1" volumeName="kubernetes.io/empty-dir/f019c85d-f532-4848-905b-b7e4480adba1-volume-directive-shadow" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179589 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179608 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10982725-01b4-4d8e-8ad4-44d43608e6b0" volumeName="kubernetes.io/projected/10982725-01b4-4d8e-8ad4-44d43608e6b0-kube-api-access-2zkkn" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179656 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="420b750b-2a29-4432-84f7-0d7709493a41" volumeName="kubernetes.io/configmap/420b750b-2a29-4432-84f7-0d7709493a41-metrics-client-ca" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179674 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44293dda-24ae-4aae-aa97-1e262c3186de" volumeName="kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-service-ca" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179690 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8411ba88-6e16-48e1-ab47-a33b4d5f0766" volumeName="kubernetes.io/configmap/8411ba88-6e16-48e1-ab47-a33b4d5f0766-config-volume" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179724 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ece532ab-396d-46f7-9de2-9e19563a5528" volumeName="kubernetes.io/projected/ece532ab-396d-46f7-9de2-9e19563a5528-kube-api-access-pdc49" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179738 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f019c85d-f532-4848-905b-b7e4480adba1" volumeName="kubernetes.io/secret/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179756 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b04324-e2ac-435b-8756-3812700e9af7" volumeName="kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179772 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d508cdb-d418-4f6c-89c5-45e02e93f9d0" volumeName="kubernetes.io/projected/1d508cdb-d418-4f6c-89c5-45e02e93f9d0-kube-api-access-jw58z" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179789 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-web-config" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179805 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8411ba88-6e16-48e1-ab47-a33b4d5f0766" volumeName="kubernetes.io/projected/8411ba88-6e16-48e1-ab47-a33b4d5f0766-kube-api-access-6cf2w" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179843 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a459d678-44f2-4eed-9cb2-f370f1dcc8c1" volumeName="kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-oauth-config" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179857 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b04324-e2ac-435b-8756-3812700e9af7" volumeName="kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-cluster-tls-config" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179872 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44293dda-24ae-4aae-aa97-1e262c3186de" volumeName="kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-oauth-config" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179889 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ddcb000-6271-4c42-a911-b3a4d14711c3" volumeName="kubernetes.io/projected/0ddcb000-6271-4c42-a911-b3a4d14711c3-kube-api-access-5h2ng" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179904 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b04324-e2ac-435b-8756-3812700e9af7" volumeName="kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-config-volume" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179937 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46932d2f-65ad-4ac4-b422-f4991d35cbd2" volumeName="kubernetes.io/projected/46932d2f-65ad-4ac4-b422-f4991d35cbd2-kube-api-access-x4p2b" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179955 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.179968 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180040 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180056 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb20209-5da1-4ef7-8625-b06c2b6493ef" volumeName="kubernetes.io/configmap/3eb20209-5da1-4ef7-8625-b06c2b6493ef-metrics-client-ca" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180072 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fe6a899-6382-488a-b52d-e20c68445e67" volumeName="kubernetes.io/empty-dir/4fe6a899-6382-488a-b52d-e20c68445e67-tmp-dir" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180126 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea8cfb3-a932-4689-8eda-f25dc44ca94a" volumeName="kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-config" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180142 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea8cfb3-a932-4689-8eda-f25dc44ca94a" volumeName="kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-trusted-ca-bundle" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180157 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180180 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19997935-7a8e-4747-bb75-4532f1a72db5" volumeName="kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-ovnkube-config" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180193 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a459d678-44f2-4eed-9cb2-f370f1dcc8c1" volumeName="kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-oauth-serving-cert" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180220 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0ac6e99-4947-4703-a31d-091a6cc9b9d7" volumeName="kubernetes.io/projected/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-kube-api-access-w4qm4" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180239 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0ac6e99-4947-4703-a31d-091a6cc9b9d7" volumeName="kubernetes.io/empty-dir/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-tuned" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180254 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44293dda-24ae-4aae-aa97-1e262c3186de" volumeName="kubernetes.io/projected/44293dda-24ae-4aae-aa97-1e262c3186de-kube-api-access-w4fnf" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180268 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0407316-563c-489c-9826-3c1372a6fbff" volumeName="kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-service-ca" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180288 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-tls-assets" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180305 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19997935-7a8e-4747-bb75-4532f1a72db5" volumeName="kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-ovnkube-script-lib" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180314 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb20209-5da1-4ef7-8625-b06c2b6493ef" volumeName="kubernetes.io/secret/3eb20209-5da1-4ef7-8625-b06c2b6493ef-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180328 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea8cfb3-a932-4689-8eda-f25dc44ca94a" volumeName="kubernetes.io/projected/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-kube-api-access-klngr" seLinuxMountContext="" Oct 30 20:27:17.180326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180352 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea8cfb3-a932-4689-8eda-f25dc44ca94a" volumeName="kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-oauth-config" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180366 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b1f827c-2d6d-4d1d-ac35-23324ff897fa" volumeName="kubernetes.io/secret/3b1f827c-2d6d-4d1d-ac35-23324ff897fa-tls-certificates" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180407 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="420b750b-2a29-4432-84f7-0d7709493a41" volumeName="kubernetes.io/secret/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180423 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-kube-api-access-f7g4d" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180435 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0407316-563c-489c-9826-3c1372a6fbff" volumeName="kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-serving-cert" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180459 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0ac6e99-4947-4703-a31d-091a6cc9b9d7" volumeName="kubernetes.io/empty-dir/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-tmp" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180475 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-grpc-tls" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180485 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0407316-563c-489c-9826-3c1372a6fbff" volumeName="kubernetes.io/projected/e0407316-563c-489c-9826-3c1372a6fbff-kube-api-access-9fkx6" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180529 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180544 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a459d678-44f2-4eed-9cb2-f370f1dcc8c1" volumeName="kubernetes.io/empty-dir/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-tmp" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180591 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ece532ab-396d-46f7-9de2-9e19563a5528" volumeName="kubernetes.io/empty-dir/ece532ab-396d-46f7-9de2-9e19563a5528-data-volume" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180622 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b04324-e2ac-435b-8756-3812700e9af7" volumeName="kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-tls-assets" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180638 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19997935-7a8e-4747-bb75-4532f1a72db5" volumeName="kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-env-overrides" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180647 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea8cfb3-a932-4689-8eda-f25dc44ca94a" volumeName="kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-service-ca" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180669 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a459d678-44f2-4eed-9cb2-f370f1dcc8c1" volumeName="kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-serving-cert" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180711 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f019c85d-f532-4848-905b-b7e4480adba1" volumeName="kubernetes.io/secret/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180742 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180791 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44293dda-24ae-4aae-aa97-1e262c3186de" volumeName="kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-trusted-ca-bundle" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180809 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46932d2f-65ad-4ac4-b422-f4991d35cbd2" volumeName="kubernetes.io/configmap/46932d2f-65ad-4ac4-b422-f4991d35cbd2-cni-binary-copy" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180826 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b04324-e2ac-435b-8756-3812700e9af7" volumeName="kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-main-db" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180862 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b04324-e2ac-435b-8756-3812700e9af7" volumeName="kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180883 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0001caf2-fffa-457e-8ab5-9574e11aced5" volumeName="kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180896 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b04324-e2ac-435b-8756-3812700e9af7" volumeName="kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-web-config" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180929 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-db" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180954 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb20209-5da1-4ef7-8625-b06c2b6493ef" volumeName="kubernetes.io/projected/3eb20209-5da1-4ef7-8625-b06c2b6493ef-kube-api-access-xzzc8" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.180974 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44293dda-24ae-4aae-aa97-1e262c3186de" volumeName="kubernetes.io/empty-dir/44293dda-24ae-4aae-aa97-1e262c3186de-tmp" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181035 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7a37dcb-96d5-4c31-a7d7-08f663036a86" volumeName="kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181055 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0407316-563c-489c-9826-3c1372a6fbff" volumeName="kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-oauth-serving-cert" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181067 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0407316-563c-489c-9826-3c1372a6fbff" volumeName="kubernetes.io/empty-dir/e0407316-563c-489c-9826-3c1372a6fbff-tmp" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181091 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb20209-5da1-4ef7-8625-b06c2b6493ef" volumeName="kubernetes.io/secret/3eb20209-5da1-4ef7-8625-b06c2b6493ef-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181102 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10982725-01b4-4d8e-8ad4-44d43608e6b0" volumeName="kubernetes.io/configmap/10982725-01b4-4d8e-8ad4-44d43608e6b0-serviceca" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181111 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config-out" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181144 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181158 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181170 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19997935-7a8e-4747-bb75-4532f1a72db5" volumeName="kubernetes.io/projected/19997935-7a8e-4747-bb75-4532f1a72db5-kube-api-access-g72wk" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181194 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b04324-e2ac-435b-8756-3812700e9af7" volumeName="kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-metrics-client-ca" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181205 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b04324-e2ac-435b-8756-3812700e9af7" volumeName="kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181219 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ece532ab-396d-46f7-9de2-9e19563a5528" volumeName="kubernetes.io/secret/ece532ab-396d-46f7-9de2-9e19563a5528-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181242 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b04324-e2ac-435b-8756-3812700e9af7" volumeName="kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-config-out" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181258 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="590e650a-39b7-4696-8bf9-c105a87fb012" volumeName="kubernetes.io/configmap/590e650a-39b7-4696-8bf9-c105a87fb012-iptables-alerter-script" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181289 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea8cfb3-a932-4689-8eda-f25dc44ca94a" volumeName="kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-oauth-serving-cert" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181314 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0407316-563c-489c-9826-3c1372a6fbff" volumeName="kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-trusted-ca-bundle" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181366 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9b04324-e2ac-435b-8756-3812700e9af7" volumeName="kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-kube-api-access-hd89k" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181417 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" volumeName="kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181451 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="420b750b-2a29-4432-84f7-0d7709493a41" volumeName="kubernetes.io/configmap/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181498 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea8cfb3-a932-4689-8eda-f25dc44ca94a" volumeName="kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-serving-cert" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181511 5755 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dff6e046-945d-4553-ba94-908dbb1a0c23" volumeName="kubernetes.io/configmap/dff6e046-945d-4553-ba94-908dbb1a0c23-konnectivity-ca" seLinuxMountContext="" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181524 5755 reconstruct.go:97] "Volume reconstruction finished" Oct 30 20:27:17.183868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.181530 5755 reconciler.go:29] "Reconciler: start to sync state" Oct 30 20:27:17.197330 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.195019 5755 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 20:27:17.204326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.204299 5755 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 20:27:17.204326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.204327 5755 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 20:27:17.204560 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.204348 5755 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 20:27:17.204560 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:17.204405 5755 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 20:27:17.211601 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.211570 5755 generic.go:356] "Generic (PLEG): container finished" podID="420b750b-2a29-4432-84f7-0d7709493a41" containerID="bc6a0881470d951a107994760dff7d53ec663177903a663e7ca9b6d242b62d8d" exitCode=0 Oct 30 20:27:17.235624 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.235592 5755 generic.go:356] "Generic (PLEG): container finished" podID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerID="ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2" exitCode=0 Oct 30 20:27:17.239096 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.239066 5755 generic.go:356] "Generic (PLEG): container finished" podID="f9b04324-e2ac-435b-8756-3812700e9af7" containerID="96f6550998f48971ac1ecf6f3bded5dd68338eec40b2f3f7df8c5567f1035af2" exitCode=0 Oct 30 20:27:17.251759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.251722 5755 generic.go:356] "Generic (PLEG): container finished" podID="d006fe2d257556fbc453c5bdc6771005" containerID="656fca1d33c016d430c96965802b5e13ad7eb2c48d2c6a7acde0d2c380e4572f" exitCode=0 Oct 30 20:27:17.266740 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.266660 5755 generic.go:356] "Generic (PLEG): container finished" podID="0ddcb000-6271-4c42-a911-b3a4d14711c3" containerID="152bbfcd3373a41f12ab164d5c5d03a85b2d039fc31541e7ee344b8f293ceaf1" exitCode=0 Oct 30 20:27:17.266740 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.266692 5755 generic.go:356] "Generic (PLEG): container finished" podID="0ddcb000-6271-4c42-a911-b3a4d14711c3" containerID="00a526f80246081311fffc5e548133db38dd5f47066079b2c06b582510a7c7c6" exitCode=0 Oct 30 20:27:17.266740 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.266704 5755 generic.go:356] "Generic (PLEG): container finished" podID="0ddcb000-6271-4c42-a911-b3a4d14711c3" containerID="0be7fd9c458c8058cbc2afca64fde921fb26d3f8d4e643101238e37847a1af72" exitCode=0 Oct 30 20:27:17.266740 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.266711 5755 generic.go:356] "Generic (PLEG): container finished" podID="0ddcb000-6271-4c42-a911-b3a4d14711c3" containerID="02c50a44279ea62be01cc7b9a9300c323ef894387058bbf8c5f8df0feec20d89" exitCode=0 Oct 30 20:27:17.266740 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.266717 5755 generic.go:356] "Generic (PLEG): container finished" podID="0ddcb000-6271-4c42-a911-b3a4d14711c3" containerID="5f66023096e606c6c1637e314d74242d533281a945118e1bc153814e5e8638db" exitCode=0 Oct 30 20:27:17.266740 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.266722 5755 generic.go:356] "Generic (PLEG): container finished" podID="0ddcb000-6271-4c42-a911-b3a4d14711c3" containerID="c2598612c921be9a11a3e45156726f45c2bd1ccdbb6e3acdeb5fa454265b7c5f" exitCode=0 Oct 30 20:27:17.268354 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:17.268331 5755 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-195.us-west-1.compute.internal\" not found" Oct 30 20:27:17.293823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.293797 5755 manager.go:324] Recovery completed Oct 30 20:27:17.304692 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:17.304657 5755 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 20:27:17.318435 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.318411 5755 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:27:17.320370 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.320346 5755 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:27:17.320500 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.320380 5755 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:27:17.320500 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.320430 5755 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:27:17.324250 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.324229 5755 cpu_manager.go:226] "Starting" policy="none" Oct 30 20:27:17.324250 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.324245 5755 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 20:27:17.324429 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.324254 5755 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:27:17.324429 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.324264 5755 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 20:27:17.324507 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.324448 5755 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 20:27:17.324507 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.324464 5755 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 20:27:17.324507 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.324488 5755 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 20:27:17.324507 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.324494 5755 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 20:27:17.324507 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.324499 5755 policy_none.go:50] "Start" Oct 30 20:27:17.324714 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.324512 5755 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 20:27:17.324714 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.324523 5755 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 20:27:17.325290 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.325271 5755 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:27:17.325374 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.325308 5755 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:27:17.325374 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.325325 5755 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:27:17.325803 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.325787 5755 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 20:27:17.325838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.325807 5755 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 20:27:17.325838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.325813 5755 policy_none.go:44] "Start" Oct 30 20:27:17.330943 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.330912 5755 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 20:27:17.333661 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.333644 5755 manager.go:341] "Starting Device Plugin manager" Oct 30 20:27:17.333769 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:17.333674 5755 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 20:27:17.333769 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.333685 5755 server.go:87] "Starting device plugin registration server" Oct 30 20:27:17.333983 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.333969 5755 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 20:27:17.334065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.333985 5755 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 20:27:17.334136 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.334095 5755 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 20:27:17.334212 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.334179 5755 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:27:17.334266 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.334219 5755 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 20:27:17.334266 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.334226 5755 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 20:27:17.335060 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.334293 5755 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:27:17.334210027Z","UUID":"7c6d5205-2521-46f5-8071-306e5109e1fc","Handler":null,"Name":"","Endpoint":""} Oct 30 20:27:17.335996 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:17.335977 5755 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 20:27:17.336094 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:17.336020 5755 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-195.us-west-1.compute.internal\" not found" Oct 30 20:27:17.336302 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.336285 5755 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 20:27:17.336363 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.336314 5755 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:27:17.358242 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.358204 5755 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:27:17.359035 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.359015 5755 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 20:27:17.419272 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.419237 5755 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:27:17.434145 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.434117 5755 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:27:17.435336 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.435313 5755 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:27:17.435448 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.435350 5755 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:27:17.435448 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.435362 5755 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-195.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:27:17.435521 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.435452 5755 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:17.458121 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.458086 5755 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:27:17.458121 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.458090 5755 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:27:17.505483 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.505438 5755 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal"] Oct 30 20:27:17.506102 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.506069 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" event={"ID":"b1b5f93e01c03391b49a05418cff0a8e","Type":"ContainerStarted","Data":"2961efb98d549f8298cc3d7048c811fcb8ee8cb4b68a1c9f4a848a899d5261c6"} Oct 30 20:27:17.506199 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.506108 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" event={"ID":"b1b5f93e01c03391b49a05418cff0a8e","Type":"ContainerStarted","Data":"81bd1aa7a4ea1b9b050060623a14ecd686bad9ce306fb444336a134c6c1c4b9d"} Oct 30 20:27:17.506199 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.506156 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" event={"ID":"d006fe2d257556fbc453c5bdc6771005","Type":"ContainerStarted","Data":"3e46fd0c758fa69a4841bcf5a5d96e5058eab27a1ccea0b2052c03e0cf7fa44c"} Oct 30 20:27:17.506199 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.506169 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" event={"ID":"d006fe2d257556fbc453c5bdc6771005","Type":"ContainerDied","Data":"656fca1d33c016d430c96965802b5e13ad7eb2c48d2c6a7acde0d2c380e4572f"} Oct 30 20:27:17.506199 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.506182 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" event={"ID":"d006fe2d257556fbc453c5bdc6771005","Type":"ContainerStarted","Data":"4b9ee59e00f9f10151f45bc9eeda5290f733fa47f3e1c8e0aa5fde6fe999f48c"} Oct 30 20:27:17.506347 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.506310 5755 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:17.506408 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.506348 5755 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:17.556323 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.556244 5755 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:17.556323 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.556320 5755 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:17.582013 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.581980 5755 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:27:17.582181 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:17.582036 5755 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:17.582181 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.581980 5755 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:27:17.582181 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:17.582121 5755 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:17.661052 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.661018 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d006fe2d257556fbc453c5bdc6771005-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"d006fe2d257556fbc453c5bdc6771005\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:17.661203 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.661057 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d006fe2d257556fbc453c5bdc6771005-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"d006fe2d257556fbc453c5bdc6771005\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:17.661203 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.661077 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b1b5f93e01c03391b49a05418cff0a8e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"b1b5f93e01c03391b49a05418cff0a8e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:17.761697 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.761660 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d006fe2d257556fbc453c5bdc6771005-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"d006fe2d257556fbc453c5bdc6771005\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:17.761697 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.761698 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d006fe2d257556fbc453c5bdc6771005-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"d006fe2d257556fbc453c5bdc6771005\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:17.761878 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.761722 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b1b5f93e01c03391b49a05418cff0a8e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"b1b5f93e01c03391b49a05418cff0a8e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:17.761878 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.761766 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b1b5f93e01c03391b49a05418cff0a8e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"b1b5f93e01c03391b49a05418cff0a8e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:17.761878 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.761762 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d006fe2d257556fbc453c5bdc6771005-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"d006fe2d257556fbc453c5bdc6771005\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:17.761878 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:17.761766 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d006fe2d257556fbc453c5bdc6771005-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal\" (UID: \"d006fe2d257556fbc453c5bdc6771005\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:18.143506 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.143469 5755 apiserver.go:52] "Watching apiserver" Oct 30 20:27:18.257666 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.257629 5755 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 20:27:18.258411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.258358 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-tw7jg","openshift-console/console-77c5f96b9d-ll5kd","openshift-image-registry/node-ca-zttcb","kube-system/global-pull-secret-syncer-lwk8k","openshift-console/console-87bf75d78-kp5hd","openshift-ovn-kubernetes/ovnkube-node-6kqkm","openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf","openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf","openshift-monitoring/prometheus-k8s-0","kube-system/konnectivity-agent-6kfsl","openshift-insights/insights-runtime-extractor-9hkvc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal","openshift-console/console-6cf86b8865-9l5xr","openshift-ingress-canary/ingress-canary-6h6z6","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q","openshift-network-operator/iptables-alerter-9mq4r","kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal","openshift-monitoring/node-exporter-qm6mr","openshift-multus/multus-additional-cni-plugins-6txlq","openshift-dns/dns-default-vgwpl","openshift-dns/node-resolver-vbtnk","openshift-cluster-node-tuning-operator/tuned-wwknf","openshift-monitoring/alertmanager-main-0","openshift-multus/multus-c4qhb","openshift-multus/network-metrics-daemon-8sd9b","openshift-console/console-7d5bb744b-ptq76"] Oct 30 20:27:18.272006 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.271980 5755 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:18.272208 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.272189 5755 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:18.329187 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.329157 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 20:27:18.332280 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.332263 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mkv2p\"" Oct 30 20:27:18.332705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.332691 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 20:27:18.332775 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.332758 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-47trg\"" Oct 30 20:27:18.335342 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.335324 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 20:27:18.335629 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.335607 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 20:27:18.335965 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.335949 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 20:27:18.336252 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.336236 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 20:27:18.336329 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.336268 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 20:27:18.336329 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.336276 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 20:27:18.336329 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.336285 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 20:27:18.336329 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.336289 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 20:27:18.336509 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.336496 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 20:27:18.336551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.336499 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 20:27:18.336551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.336515 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 20:27:18.336551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.336513 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 20:27:18.336551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.336547 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:27:18.336780 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.336767 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 20:27:18.336816 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.336769 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 20:27:18.336860 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.336818 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 20:27:18.336994 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.336982 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 20:27:18.337033 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.337015 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 20:27:18.337033 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.337020 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 20:27:18.338010 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.337992 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 20:27:18.338010 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.337996 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 20:27:18.338751 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.338740 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 20:27:18.338802 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.338792 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 20:27:18.338942 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.338927 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 20:27:18.339028 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.339010 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 20:27:18.339079 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.339026 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 20:27:18.339113 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.339077 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:27:18.339113 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.339081 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 20:27:18.365345 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.365312 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-serving-cert\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:18.365345 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.365348 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46932d2f-65ad-4ac4-b422-f4991d35cbd2-cni-binary-copy\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.365561 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.365374 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-netns\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.365561 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.365415 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-cni-netd\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.365561 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.365438 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-kubernetes\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.365561 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.365492 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-var-lib-kubelet\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.365561 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.365524 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dff6e046-945d-4553-ba94-908dbb1a0c23-agent-certs\") pod \"konnectivity-agent-6kfsl\" (UID: \"dff6e046-945d-4553-ba94-908dbb1a0c23\") " pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:27:18.365802 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.365587 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/590e650a-39b7-4696-8bf9-c105a87fb012-iptables-alerter-script\") pod \"iptables-alerter-9mq4r\" (UID: \"590e650a-39b7-4696-8bf9-c105a87fb012\") " pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:27:18.365802 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.365638 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/590e650a-39b7-4696-8bf9-c105a87fb012-host-slash\") pod \"iptables-alerter-9mq4r\" (UID: \"590e650a-39b7-4696-8bf9-c105a87fb012\") " pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:27:18.365802 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.365720 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46932d2f-65ad-4ac4-b422-f4991d35cbd2-cni-binary-copy\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.365802 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.365732 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-run-netns\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.365802 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.365784 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.365976 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.365823 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-host\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.365976 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.365857 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4qm4\" (UniqueName: \"kubernetes.io/projected/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-kube-api-access-w4qm4\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.365976 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.365856 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/590e650a-39b7-4696-8bf9-c105a87fb012-iptables-alerter-script\") pod \"iptables-alerter-9mq4r\" (UID: \"590e650a-39b7-4696-8bf9-c105a87fb012\") " pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:27:18.365976 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.365911 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.366080 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366055 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-multus-certs\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.366174 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366150 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.366213 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366184 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-systemd\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.366261 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366217 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-socket-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.366261 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366252 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.366351 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366279 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.366351 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366333 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.366456 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366423 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2ng\" (UniqueName: \"kubernetes.io/projected/0ddcb000-6271-4c42-a911-b3a4d14711c3-kube-api-access-5h2ng\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.366500 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366470 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ece532ab-396d-46f7-9de2-9e19563a5528-data-volume\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:27:18.366500 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366487 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.366593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366568 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-service-ca\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:18.366593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366579 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ece532ab-396d-46f7-9de2-9e19563a5528-data-volume\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:27:18.366690 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366602 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9whs\" (UniqueName: \"kubernetes.io/projected/590e650a-39b7-4696-8bf9-c105a87fb012-kube-api-access-w9whs\") pod \"iptables-alerter-9mq4r\" (UID: \"590e650a-39b7-4696-8bf9-c105a87fb012\") " pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:27:18.366690 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366627 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-var-lib-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.366769 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366692 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-etc-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.366805 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366775 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe6a899-6382-488a-b52d-e20c68445e67-tmp-dir\") pod \"node-resolver-vbtnk\" (UID: \"4fe6a899-6382-488a-b52d-e20c68445e67\") " pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:27:18.366835 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366803 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ece532ab-396d-46f7-9de2-9e19563a5528-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:27:18.366871 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366849 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8411ba88-6e16-48e1-ab47-a33b4d5f0766-metrics-tls\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:27:18.366905 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366889 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-cni-multus\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.366941 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366892 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe6a899-6382-488a-b52d-e20c68445e67-tmp-dir\") pod \"node-resolver-vbtnk\" (UID: \"4fe6a899-6382-488a-b52d-e20c68445e67\") " pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:27:18.366941 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366930 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-ovn\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.367024 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366961 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-lib-modules\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.367024 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.366984 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbs9\" (UniqueName: \"kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9\") pod \"network-check-target-tw7jg\" (UID: \"0001caf2-fffa-457e-8ab5-9574e11aced5\") " pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:27:18.367087 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367059 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ece532ab-396d-46f7-9de2-9e19563a5528-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:27:18.367121 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367090 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.367121 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367114 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdc49\" (UniqueName: \"kubernetes.io/projected/ece532ab-396d-46f7-9de2-9e19563a5528-kube-api-access-pdc49\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:27:18.367192 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367145 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95f11ca1-5d71-4135-bf16-0fe6c24706c0-dbus\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:27:18.367192 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367173 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-config\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:18.367283 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367197 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-systemd-units\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.367283 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367230 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-log-socket\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.367363 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367280 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-tuned\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.367363 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367312 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10982725-01b4-4d8e-8ad4-44d43608e6b0-host\") pod \"node-ca-zttcb\" (UID: \"10982725-01b4-4d8e-8ad4-44d43608e6b0\") " pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:27:18.367363 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367318 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ddcb000-6271-4c42-a911-b3a4d14711c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.367363 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367337 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-cni-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.367363 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367361 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:27:18.367583 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367370 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-tuned\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.367583 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367409 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-tmp\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.367583 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367443 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-kubelet\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.367583 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367462 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-ovnkube-config\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.367583 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367478 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-tmp\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.367583 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367503 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-ovnkube-script-lib\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.367583 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367558 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysconfig\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.367918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367594 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-sys-fs\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.367918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367637 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdrp4\" (UniqueName: \"kubernetes.io/projected/6a305968-1cca-480f-9360-aeb99c0ef2de-kube-api-access-pdrp4\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.367918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367657 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-ovnkube-config\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.367918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367663 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10982725-01b4-4d8e-8ad4-44d43608e6b0-serviceca\") pod \"node-ca-zttcb\" (UID: \"10982725-01b4-4d8e-8ad4-44d43608e6b0\") " pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:27:18.367918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367684 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4p2b\" (UniqueName: \"kubernetes.io/projected/46932d2f-65ad-4ac4-b422-f4991d35cbd2-kube-api-access-x4p2b\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.367918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367712 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-daemon-config\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.367918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367732 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-ovnkube-script-lib\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.367918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367777 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-modprobe-d\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.367918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367804 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zkkn\" (UniqueName: \"kubernetes.io/projected/10982725-01b4-4d8e-8ad4-44d43608e6b0-kube-api-access-2zkkn\") pod \"node-ca-zttcb\" (UID: \"10982725-01b4-4d8e-8ad4-44d43608e6b0\") " pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:27:18.367918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367829 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4fe6a899-6382-488a-b52d-e20c68445e67-hosts-file\") pod \"node-resolver-vbtnk\" (UID: \"4fe6a899-6382-488a-b52d-e20c68445e67\") " pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:27:18.367918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367866 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ece532ab-396d-46f7-9de2-9e19563a5528-crio-socket\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:27:18.367918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367893 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8411ba88-6e16-48e1-ab47-a33b4d5f0766-config-volume\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367924 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-daemon-config\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367928 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8411ba88-6e16-48e1-ab47-a33b4d5f0766-tmp-dir\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367956 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-oauth-serving-cert\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367975 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-slash\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367988 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8411ba88-6e16-48e1-ab47-a33b4d5f0766-tmp-dir\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.367989 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-device-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368047 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-tmp\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368084 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc7c8\" (UniqueName: \"kubernetes.io/projected/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-kube-api-access-xc7c8\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368118 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-cni-bin\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368132 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-tmp\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368152 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cf2w\" (UniqueName: \"kubernetes.io/projected/8411ba88-6e16-48e1-ab47-a33b4d5f0766-kube-api-access-6cf2w\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368178 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-cni-bin\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368205 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d508cdb-d418-4f6c-89c5-45e02e93f9d0-cert\") pod \"ingress-canary-6h6z6\" (UID: \"1d508cdb-d418-4f6c-89c5-45e02e93f9d0\") " pod="openshift-ingress-canary/ingress-canary-6h6z6" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368236 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wz4p\" (UniqueName: \"kubernetes.io/projected/4fe6a899-6382-488a-b52d-e20c68445e67-kube-api-access-9wz4p\") pod \"node-resolver-vbtnk\" (UID: \"4fe6a899-6382-488a-b52d-e20c68445e67\") " pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368311 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ece532ab-396d-46f7-9de2-9e19563a5528-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368344 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95f11ca1-5d71-4135-bf16-0fe6c24706c0-kubelet-config\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368362 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-k8s-cni-cncf-io\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368379 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3b1f827c-2d6d-4d1d-ac35-23324ff897fa-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5kk8q\" (UID: \"3b1f827c-2d6d-4d1d-ac35-23324ff897fa\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368376 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10982725-01b4-4d8e-8ad4-44d43608e6b0-serviceca\") pod \"node-ca-zttcb\" (UID: \"10982725-01b4-4d8e-8ad4-44d43608e6b0\") " pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368416 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-877qg\" (UniqueName: \"kubernetes.io/projected/d7a37dcb-96d5-4c31-a7d7-08f663036a86-kube-api-access-877qg\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368439 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19997935-7a8e-4747-bb75-4532f1a72db5-ovn-node-metrics-cert\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368464 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g72wk\" (UniqueName: \"kubernetes.io/projected/19997935-7a8e-4747-bb75-4532f1a72db5-kube-api-access-g72wk\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368482 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-registration-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368499 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-oauth-config\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:18.368551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368566 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-socket-dir-parent\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.369467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368595 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-kubelet\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.369467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368674 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19997935-7a8e-4747-bb75-4532f1a72db5-ovn-node-metrics-cert\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.369467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368817 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-conf-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.369467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368838 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-etc-kubernetes\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.369467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368856 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-systemd\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.369467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368870 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.369467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.368889 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-hostroot\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.369467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369033 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-env-overrides\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.369467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369056 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysctl-conf\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.369467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369070 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-sys\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.369467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369087 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw58z\" (UniqueName: \"kubernetes.io/projected/1d508cdb-d418-4f6c-89c5-45e02e93f9d0-kube-api-access-jw58z\") pod \"ingress-canary-6h6z6\" (UID: \"1d508cdb-d418-4f6c-89c5-45e02e93f9d0\") " pod="openshift-ingress-canary/ingress-canary-6h6z6" Oct 30 20:27:18.369467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369219 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19997935-7a8e-4747-bb75-4532f1a72db5-env-overrides\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.369467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369248 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-system-cni-dir\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.369467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369325 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:27:18.369467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369366 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-system-cni-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.369467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369413 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-cnibin\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.369977 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369502 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-node-log\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.369977 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369537 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysctl-d\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.369977 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369560 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-run\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.369977 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369578 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.369977 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369593 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-etc-selinux\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.369977 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369637 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-cnibin\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.369977 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369677 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dff6e046-945d-4553-ba94-908dbb1a0c23-konnectivity-ca\") pod \"konnectivity-agent-6kfsl\" (UID: \"dff6e046-945d-4553-ba94-908dbb1a0c23\") " pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:27:18.369977 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.369852 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dff6e046-945d-4553-ba94-908dbb1a0c23-konnectivity-ca\") pod \"konnectivity-agent-6kfsl\" (UID: \"dff6e046-945d-4553-ba94-908dbb1a0c23\") " pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:27:18.370927 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.370913 5755 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 20:27:18.387012 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.386986 5755 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:27:18.387123 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:18.387036 5755 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:18.387236 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.387218 5755 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:27:18.387267 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:18.387260 5755 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" Oct 30 20:27:18.423770 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.423681 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 20:27:18.423923 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.423823 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 20:27:18.423923 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.423845 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 20:27:18.424198 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.424179 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:27:18.424353 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.424274 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 20:27:18.424466 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.424275 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 20:27:18.424550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.424533 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:27:18.424653 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.424636 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 20:27:18.424735 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.424717 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 20:27:18.424789 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.424721 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 20:27:18.424939 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.424913 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 20:27:18.425036 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.424948 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 20:27:18.425102 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.425031 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 20:27:18.428352 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.428327 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8411ba88-6e16-48e1-ab47-a33b4d5f0766-config-volume\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:27:18.428475 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.428378 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-oauth-serving-cert\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:18.428663 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.428637 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-service-ca\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:18.429336 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.429311 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-config\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:18.471115 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471076 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44293dda-24ae-4aae-aa97-1e262c3186de-tmp\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:18.471115 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471117 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-netns\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.471317 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471181 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-cni-netd\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.471317 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471215 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-kubernetes\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.471317 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471222 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44293dda-24ae-4aae-aa97-1e262c3186de-tmp\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:18.471317 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471235 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.471525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471327 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-netns\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.471525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471346 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-kubernetes\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.471525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471353 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-run-netns\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.471525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471407 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-cni-netd\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.471525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471412 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.471525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471440 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.471525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471447 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:18.471525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471416 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-run-netns\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.471525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471361 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.471525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471485 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb20209-5da1-4ef7-8625-b06c2b6493ef-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:18.471525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471528 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-oauth-serving-cert\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471548 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-systemd\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471569 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-serving-cert\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471584 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-oauth-config\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471594 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-systemd\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471599 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471616 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7g4d\" (UniqueName: \"kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-kube-api-access-f7g4d\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471635 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-var-lib-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471661 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-oauth-serving-cert\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471686 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471693 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-var-lib-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471709 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-cni-multus\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471750 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-cni-multus\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471750 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-oauth-serving-cert\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471752 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-ovn\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471788 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-ovn\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471806 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471843 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-oauth-serving-cert\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471869 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f019c85d-f532-4848-905b-b7e4480adba1-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471892 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-oauth-serving-cert\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471947 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89kxn\" (UniqueName: \"kubernetes.io/projected/f019c85d-f532-4848-905b-b7e4480adba1-kube-api-access-89kxn\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.471982 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-systemd-units\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472002 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-oauth-serving-cert\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472011 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzzc8\" (UniqueName: \"kubernetes.io/projected/3eb20209-5da1-4ef7-8625-b06c2b6493ef-kube-api-access-xzzc8\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:18.472029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472037 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472063 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472039 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-systemd-units\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472091 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472126 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/420b750b-2a29-4432-84f7-0d7709493a41-metrics-client-ca\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472157 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-cni-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472183 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-trusted-ca-bundle\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472201 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472216 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472236 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-cni-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472243 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-kubelet\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472252 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472278 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysconfig\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472303 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-sys-fs\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472331 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-service-ca\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472342 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-sys-fs\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472348 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysconfig\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472293 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-kubelet\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472357 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-modprobe-d\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472420 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4fe6a899-6382-488a-b52d-e20c68445e67-hosts-file\") pod \"node-resolver-vbtnk\" (UID: \"4fe6a899-6382-488a-b52d-e20c68445e67\") " pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472440 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472460 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fkx6\" (UniqueName: \"kubernetes.io/projected/e0407316-563c-489c-9826-3c1372a6fbff-kube-api-access-9fkx6\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472476 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-tmp\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472479 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-service-ca\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472496 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472499 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-modprobe-d\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472524 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472537 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-tmp\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472537 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4fe6a899-6382-488a-b52d-e20c68445e67-hosts-file\") pod \"node-resolver-vbtnk\" (UID: \"4fe6a899-6382-488a-b52d-e20c68445e67\") " pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472557 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-config-volume\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472575 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3eb20209-5da1-4ef7-8625-b06c2b6493ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472593 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-sys\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472619 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472651 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472678 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-console-config\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472688 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472699 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-trusted-ca-bundle\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472724 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472749 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472770 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-cni-bin\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472802 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95f11ca1-5d71-4135-bf16-0fe6c24706c0-kubelet-config\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472821 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-cni-bin\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472835 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-oauth-config\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472849 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95f11ca1-5d71-4135-bf16-0fe6c24706c0-kubelet-config\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472851 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-serving-cert\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472882 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472905 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-k8s-cni-cncf-io\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472925 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-registration-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472944 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-socket-dir-parent\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472962 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-etc-kubernetes\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472988 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-systemd\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472996 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-k8s-cni-cncf-io\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.472999 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-registration-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473019 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-socket-dir-parent\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.473001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473013 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473022 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-etc-kubernetes\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473052 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-systemd\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473061 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-system-cni-dir\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473094 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473054 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473122 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-console-config\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473168 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-system-cni-dir\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473203 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-hostroot\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473223 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysctl-conf\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473238 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-console-config\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473240 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473260 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-hostroot\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473270 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-console-config\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473308 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-cnibin\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473332 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-node-log\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473351 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-run\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473377 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-etc-selinux\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473415 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-cnibin\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473420 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-oauth-config\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473422 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysctl-conf\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473444 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-node-log\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473453 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-run\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473456 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-var-lib-kubelet\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473488 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-service-ca\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473495 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-var-lib-kubelet\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473506 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-accelerators-collector-config\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473510 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-etc-selinux\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473526 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nn8g\" (UniqueName: \"kubernetes.io/projected/420b750b-2a29-4432-84f7-0d7709493a41-kube-api-access-4nn8g\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473545 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/590e650a-39b7-4696-8bf9-c105a87fb012-host-slash\") pod \"iptables-alerter-9mq4r\" (UID: \"590e650a-39b7-4696-8bf9-c105a87fb012\") " pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473560 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-host\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473578 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473606 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/590e650a-39b7-4696-8bf9-c105a87fb012-host-slash\") pod \"iptables-alerter-9mq4r\" (UID: \"590e650a-39b7-4696-8bf9-c105a87fb012\") " pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473597 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-service-ca\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473623 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-service-ca\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473684 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-multus-certs\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473690 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-host\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473712 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473735 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-accelerators-collector-config\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473753 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-service-ca\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473738 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-socket-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473769 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-run-multus-certs\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473784 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-run-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473787 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473809 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-socket-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473815 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473829 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-config\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473854 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-etc-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473873 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klngr\" (UniqueName: \"kubernetes.io/projected/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-kube-api-access-klngr\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473917 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-etc-openvswitch\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.473947 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-wtmp\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474022 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-lib-modules\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474042 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95f11ca1-5d71-4135-bf16-0fe6c24706c0-dbus\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474066 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-log-socket\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474082 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10982725-01b4-4d8e-8ad4-44d43608e6b0-host\") pod \"node-ca-zttcb\" (UID: \"10982725-01b4-4d8e-8ad4-44d43608e6b0\") " pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474102 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474132 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95f11ca1-5d71-4135-bf16-0fe6c24706c0-dbus\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474134 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-log-socket\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474132 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-lib-modules\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474130 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-tls\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474204 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474227 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474243 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd89k\" (UniqueName: \"kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-kube-api-access-hd89k\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474204 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10982725-01b4-4d8e-8ad4-44d43608e6b0-host\") pod \"node-ca-zttcb\" (UID: \"10982725-01b4-4d8e-8ad4-44d43608e6b0\") " pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474265 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474295 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ece532ab-396d-46f7-9de2-9e19563a5528-crio-socket\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474321 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-web-config\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474349 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-root\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474365 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ece532ab-396d-46f7-9de2-9e19563a5528-crio-socket\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474377 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-slash\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474435 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-slash\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474455 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-device-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474486 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474513 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474533 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-device-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474536 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474562 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-cni-bin\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474594 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474599 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eb20209-5da1-4ef7-8625-b06c2b6493ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474624 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0407316-563c-489c-9826-3c1372a6fbff-tmp\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474631 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19997935-7a8e-4747-bb75-4532f1a72db5-host-cni-bin\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474651 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-config\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474656 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474684 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0407316-563c-489c-9826-3c1372a6fbff-tmp\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474712 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-config-out\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474750 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474781 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474801 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-config-out\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474820 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4fnf\" (UniqueName: \"kubernetes.io/projected/44293dda-24ae-4aae-aa97-1e262c3186de-kube-api-access-w4fnf\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474849 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f019c85d-f532-4848-905b-b7e4480adba1-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474886 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474912 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474924 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474948 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-kubelet\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474970 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.474980 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-conf-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475025 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-textfile\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475039 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-multus-conf-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475042 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-host-var-lib-kubelet\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475055 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-sys\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475075 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f019c85d-f532-4848-905b-b7e4480adba1-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475095 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-serving-cert\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475104 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-sys\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475125 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-trusted-ca-bundle\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475144 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-system-cni-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475164 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysctl-d\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475172 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-textfile\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475193 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475221 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46932d2f-65ad-4ac4-b422-f4991d35cbd2-system-cni-dir\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475222 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-cnibin\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475250 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ddcb000-6271-4c42-a911-b3a4d14711c3-cnibin\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475317 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-etc-sysctl-d\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.475331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475316 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a305968-1cca-480f-9360-aeb99c0ef2de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475592 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475592 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475616 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.475681 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476001 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476186 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9whs\" (UniqueName: \"kubernetes.io/projected/590e650a-39b7-4696-8bf9-c105a87fb012-kube-api-access-w9whs\") pod \"iptables-alerter-9mq4r\" (UID: \"590e650a-39b7-4696-8bf9-c105a87fb012\") " pod="openshift-network-operator/iptables-alerter-9mq4r" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476283 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476323 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476367 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476409 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476428 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476433 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476463 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476483 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476489 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476502 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476504 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476535 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476552 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476540 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476576 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476620 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476635 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476675 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476733 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-52gtb\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476811 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476810 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476874 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476888 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nwr4g\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.476954 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9wdmm\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477056 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477196 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdc49\" (UniqueName: \"kubernetes.io/projected/ece532ab-396d-46f7-9de2-9e19563a5528-kube-api-access-pdc49\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477247 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4bqw6\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477323 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477324 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-877qg\" (UniqueName: \"kubernetes.io/projected/d7a37dcb-96d5-4c31-a7d7-08f663036a86-kube-api-access-877qg\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477363 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bpqv7\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477364 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477430 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477482 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477491 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4qm4\" (UniqueName: \"kubernetes.io/projected/e0ac6e99-4947-4703-a31d-091a6cc9b9d7-kube-api-access-w4qm4\") pod \"tuned-wwknf\" (UID: \"e0ac6e99-4947-4703-a31d-091a6cc9b9d7\") " pod="openshift-cluster-node-tuning-operator/tuned-wwknf" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477503 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-78mnk\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477514 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477563 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wczr5\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477621 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ld6lr\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477663 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d9pbw\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477729 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477742 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5tplx\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477727 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cnj9l\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477885 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cf2w\" (UniqueName: \"kubernetes.io/projected/8411ba88-6e16-48e1-ab47-a33b4d5f0766-kube-api-access-6cf2w\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477920 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4p2b\" (UniqueName: \"kubernetes.io/projected/46932d2f-65ad-4ac4-b422-f4991d35cbd2-kube-api-access-x4p2b\") pod \"multus-c4qhb\" (UID: \"46932d2f-65ad-4ac4-b422-f4991d35cbd2\") " pod="openshift-multus/multus-c4qhb" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.477934 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478002 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478079 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw58z\" (UniqueName: \"kubernetes.io/projected/1d508cdb-d418-4f6c-89c5-45e02e93f9d0-kube-api-access-jw58z\") pod \"ingress-canary-6h6z6\" (UID: \"1d508cdb-d418-4f6c-89c5-45e02e93f9d0\") " pod="openshift-ingress-canary/ingress-canary-6h6z6" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478088 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zkkn\" (UniqueName: \"kubernetes.io/projected/10982725-01b4-4d8e-8ad4-44d43608e6b0-kube-api-access-2zkkn\") pod \"node-ca-zttcb\" (UID: \"10982725-01b4-4d8e-8ad4-44d43608e6b0\") " pod="openshift-image-registry/node-ca-zttcb" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478139 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6w4b8\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478248 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478298 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h2ng\" (UniqueName: \"kubernetes.io/projected/0ddcb000-6271-4c42-a911-b3a4d14711c3-kube-api-access-5h2ng\") pod \"multus-additional-cni-plugins-6txlq\" (UID: \"0ddcb000-6271-4c42-a911-b3a4d14711c3\") " pod="openshift-multus/multus-additional-cni-plugins-6txlq" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478328 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-94phh\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478348 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g72wk\" (UniqueName: \"kubernetes.io/projected/19997935-7a8e-4747-bb75-4532f1a72db5-kube-api-access-g72wk\") pod \"ovnkube-node-6kqkm\" (UID: \"19997935-7a8e-4747-bb75-4532f1a72db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478405 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478441 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lwgrd\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478482 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nnmlc\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478528 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc7c8\" (UniqueName: \"kubernetes.io/projected/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-kube-api-access-xc7c8\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478549 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478562 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d508cdb-d418-4f6c-89c5-45e02e93f9d0-cert\") pod \"ingress-canary-6h6z6\" (UID: \"1d508cdb-d418-4f6c-89c5-45e02e93f9d0\") " pod="openshift-ingress-canary/ingress-canary-6h6z6" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478598 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-chbcw\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478596 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ece532ab-396d-46f7-9de2-9e19563a5528-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hkvc\" (UID: \"ece532ab-396d-46f7-9de2-9e19563a5528\") " pod="openshift-insights/insights-runtime-extractor-9hkvc" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478682 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478702 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3b1f827c-2d6d-4d1d-ac35-23324ff897fa-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5kk8q\" (UID: \"3b1f827c-2d6d-4d1d-ac35-23324ff897fa\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478733 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-oauth-config\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478839 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bx2zg\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478894 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478930 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qmclr\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.478983 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdrp4\" (UniqueName: \"kubernetes.io/projected/6a305968-1cca-480f-9360-aeb99c0ef2de-kube-api-access-pdrp4\") pod \"aws-ebs-csi-driver-node-wfcvw\" (UID: \"6a305968-1cca-480f-9360-aeb99c0ef2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfcvw" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.479094 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fk349e28vqu0g\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.479142 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fngvs\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.479218 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bxjhb\"" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.479325 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wz4p\" (UniqueName: \"kubernetes.io/projected/4fe6a899-6382-488a-b52d-e20c68445e67-kube-api-access-9wz4p\") pod \"node-resolver-vbtnk\" (UID: \"4fe6a899-6382-488a-b52d-e20c68445e67\") " pod="openshift-dns/node-resolver-vbtnk" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.479601 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgbs9\" (UniqueName: \"kubernetes.io/projected/0001caf2-fffa-457e-8ab5-9574e11aced5-kube-api-access-wgbs9\") pod \"network-check-target-tw7jg\" (UID: \"0001caf2-fffa-457e-8ab5-9574e11aced5\") " pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:27:18.480331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.479921 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95f11ca1-5d71-4135-bf16-0fe6c24706c0-original-pull-secret\") pod \"global-pull-secret-syncer-lwk8k\" (UID: \"95f11ca1-5d71-4135-bf16-0fe6c24706c0\") " pod="kube-system/global-pull-secret-syncer-lwk8k" Oct 30 20:27:18.482771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.482559 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:18.482771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.482602 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.482771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.482575 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-serving-cert\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:18.482771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.482612 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb20209-5da1-4ef7-8625-b06c2b6493ef-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:18.482771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.482643 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.482771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.482693 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-oauth-config\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:18.482771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.482708 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-trusted-ca-bundle\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:18.482771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.482593 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/420b750b-2a29-4432-84f7-0d7709493a41-metrics-client-ca\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.482771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.482759 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-config-volume\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.482771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.482770 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.483268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.482775 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:18.483268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.482827 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f019c85d-f532-4848-905b-b7e4480adba1-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:18.483268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.482859 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.483268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.482929 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3eb20209-5da1-4ef7-8625-b06c2b6493ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:18.483268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.482972 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.483268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.483010 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-serving-cert\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:18.483268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.483063 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.483268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.483140 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.483268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.483068 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-oauth-config\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:18.483268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.483138 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.483268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.483166 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.483268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.483254 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.483670 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.483363 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.483786 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.483769 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.483839 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.483788 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-trusted-ca-bundle\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:18.483941 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.483926 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-oauth-config\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:18.483975 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.483936 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f019c85d-f532-4848-905b-b7e4480adba1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:18.483975 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.483966 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.484653 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.484634 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.484744 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.484650 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-web-config\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.484744 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.484640 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.484744 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.484675 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-tls\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.484744 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.484675 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.484897 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.484762 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.484897 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.484792 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eb20209-5da1-4ef7-8625-b06c2b6493ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:18.484897 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.484835 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.484897 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.484889 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.485006 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.484991 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.486070 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.486044 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-serving-cert\") pod \"console-7d5bb744b-ptq76\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:18.486070 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.486058 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-serving-cert\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:18.486187 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.486057 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dff6e046-945d-4553-ba94-908dbb1a0c23-agent-certs\") pod \"konnectivity-agent-6kfsl\" (UID: \"dff6e046-945d-4553-ba94-908dbb1a0c23\") " pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:27:18.486187 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.486150 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.486245 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.486193 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.486245 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.486223 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-trusted-ca-bundle\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:18.487328 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.487311 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8411ba88-6e16-48e1-ab47-a33b4d5f0766-metrics-tls\") pod \"dns-default-vgwpl\" (UID: \"8411ba88-6e16-48e1-ab47-a33b4d5f0766\") " pod="openshift-dns/dns-default-vgwpl" Oct 30 20:27:18.488091 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.488075 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.488227 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.488215 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a37dcb-96d5-4c31-a7d7-08f663036a86-metrics-certs\") pod \"network-metrics-daemon-8sd9b\" (UID: \"d7a37dcb-96d5-4c31-a7d7-08f663036a86\") " pod="openshift-multus/network-metrics-daemon-8sd9b" Oct 30 20:27:18.537855 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.537816 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4fnf\" (UniqueName: \"kubernetes.io/projected/44293dda-24ae-4aae-aa97-1e262c3186de-kube-api-access-w4fnf\") pod \"console-6cf86b8865-9l5xr\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:18.538181 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.538158 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7g4d\" (UniqueName: \"kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-kube-api-access-f7g4d\") pod \"prometheus-k8s-0\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:18.538259 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.538169 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzzc8\" (UniqueName: \"kubernetes.io/projected/3eb20209-5da1-4ef7-8625-b06c2b6493ef-kube-api-access-xzzc8\") pod \"openshift-state-metrics-7bd7489957-tpsmf\" (UID: \"3eb20209-5da1-4ef7-8625-b06c2b6493ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tpsmf" Oct 30 20:27:18.538493 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.538472 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fkx6\" (UniqueName: \"kubernetes.io/projected/e0407316-563c-489c-9826-3c1372a6fbff-kube-api-access-9fkx6\") pod \"console-87bf75d78-kp5hd\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:18.538726 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.538709 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89kxn\" (UniqueName: \"kubernetes.io/projected/f019c85d-f532-4848-905b-b7e4480adba1-kube-api-access-89kxn\") pod \"kube-state-metrics-5dc8d7d89b-ffxbf\" (UID: \"f019c85d-f532-4848-905b-b7e4480adba1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ffxbf" Oct 30 20:27:18.538898 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.538883 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klngr\" (UniqueName: \"kubernetes.io/projected/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-kube-api-access-klngr\") pod \"console-77c5f96b9d-ll5kd\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:18.539330 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.539311 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nn8g\" (UniqueName: \"kubernetes.io/projected/420b750b-2a29-4432-84f7-0d7709493a41-kube-api-access-4nn8g\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.545424 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.545404 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd89k\" (UniqueName: \"kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-kube-api-access-hd89k\") pod \"alertmanager-main-0\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:18.575944 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.575515 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-sys\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.575944 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.575559 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-wtmp\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.575944 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.575594 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-root\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.575944 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.575611 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-sys\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.575944 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.575641 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-root\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.575944 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.575668 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/420b750b-2a29-4432-84f7-0d7709493a41-node-exporter-wtmp\") pod \"node-exporter-qm6mr\" (UID: \"420b750b-2a29-4432-84f7-0d7709493a41\") " pod="openshift-monitoring/node-exporter-qm6mr" Oct 30 20:27:18.576142 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.576120 5755 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:27:18.820880 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.820847 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d5bb744b-ptq76"] Oct 30 20:27:18.823050 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:18.823031 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vgwpl" Oct 30 20:27:19.905214 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:19.905175 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:20.045743 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:20.045704 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:20.051180 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:20.051147 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:20.447943 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:20.447907 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:27:21.285686 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:21.285650 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f928cc6-d12f-42d6-82d3-4673ed8a9d07","Type":"ContainerStarted","Data":"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c"} Oct 30 20:27:21.286166 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:21.285718 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f928cc6-d12f-42d6-82d3-4673ed8a9d07","Type":"ContainerStarted","Data":"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f"} Oct 30 20:27:21.286166 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:21.285745 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f928cc6-d12f-42d6-82d3-4673ed8a9d07","Type":"ContainerStarted","Data":"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09"} Oct 30 20:27:21.286166 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:21.285802 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f928cc6-d12f-42d6-82d3-4673ed8a9d07","Type":"ContainerStarted","Data":"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d"} Oct 30 20:27:22.289160 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:22.289132 5755 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:27:22.668535 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:22.668442 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:22.684679 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:22.684655 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:22.783772 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:22.783741 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5kk8q" Oct 30 20:27:25.366528 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:25.366485 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:27.821069 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:27.821034 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:27.821465 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:27.821237 5755 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:27:28.000533 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:28.000490 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:29.954503 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:29.954466 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:38.005901 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.005867 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:27:38.009547 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.009520 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:27:38.009838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.009802 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="kube-rbac-proxy-web" containerID="cri-o://45619b244dae9537e65d1c12498cac559619b243fb5f12ac0754454a4d154c89" gracePeriod=120 Oct 30 20:27:38.009838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.009806 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="alertmanager" containerID="cri-o://5c0d7597c7e82e7d2f0db173ad6ae4effb8be243d563b2b64aa1155a98aaa495" gracePeriod=120 Oct 30 20:27:38.010015 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.009860 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="config-reloader" containerID="cri-o://b23bf95a50b4927953fb449472b426f4708434cbcc47cd8cf8ac35763f3f3a31" gracePeriod=120 Oct 30 20:27:38.010015 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.009817 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="kube-rbac-proxy" containerID="cri-o://4ed423139336b82a159517f2298c1a45926f7e6997a830236453c194619d9e5c" gracePeriod=120 Oct 30 20:27:38.010015 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.009882 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="kube-rbac-proxy-metric" containerID="cri-o://4e2ea3b15932e039201023f2710e7c35771606421e5173c96d40ce08a46a8a58" gracePeriod=120 Oct 30 20:27:38.010015 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.009817 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="prom-label-proxy" containerID="cri-o://02959c2b916e35e40644a616fd4bd315a1266f7b98d5f8b77e8bc1d8abbc370b" gracePeriod=120 Oct 30 20:27:38.211314 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.211288 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-57db778948-tm9wd"] Oct 30 20:27:38.227171 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.227150 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.273091 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.273013 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57db778948-tm9wd"] Oct 30 20:27:38.351757 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.351729 5755 generic.go:356] "Generic (PLEG): container finished" podID="f9b04324-e2ac-435b-8756-3812700e9af7" containerID="02959c2b916e35e40644a616fd4bd315a1266f7b98d5f8b77e8bc1d8abbc370b" exitCode=0 Oct 30 20:27:38.351894 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.351752 5755 generic.go:356] "Generic (PLEG): container finished" podID="f9b04324-e2ac-435b-8756-3812700e9af7" containerID="4e2ea3b15932e039201023f2710e7c35771606421e5173c96d40ce08a46a8a58" exitCode=0 Oct 30 20:27:38.351894 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.351782 5755 generic.go:356] "Generic (PLEG): container finished" podID="f9b04324-e2ac-435b-8756-3812700e9af7" containerID="4ed423139336b82a159517f2298c1a45926f7e6997a830236453c194619d9e5c" exitCode=0 Oct 30 20:27:38.351894 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.351790 5755 generic.go:356] "Generic (PLEG): container finished" podID="f9b04324-e2ac-435b-8756-3812700e9af7" containerID="b23bf95a50b4927953fb449472b426f4708434cbcc47cd8cf8ac35763f3f3a31" exitCode=0 Oct 30 20:27:38.351894 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.351796 5755 generic.go:356] "Generic (PLEG): container finished" podID="f9b04324-e2ac-435b-8756-3812700e9af7" containerID="5c0d7597c7e82e7d2f0db173ad6ae4effb8be243d563b2b64aa1155a98aaa495" exitCode=0 Oct 30 20:27:38.351894 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.351808 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9b04324-e2ac-435b-8756-3812700e9af7","Type":"ContainerDied","Data":"02959c2b916e35e40644a616fd4bd315a1266f7b98d5f8b77e8bc1d8abbc370b"} Oct 30 20:27:38.351894 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.351869 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9b04324-e2ac-435b-8756-3812700e9af7","Type":"ContainerDied","Data":"4e2ea3b15932e039201023f2710e7c35771606421e5173c96d40ce08a46a8a58"} Oct 30 20:27:38.351894 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.351894 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9b04324-e2ac-435b-8756-3812700e9af7","Type":"ContainerDied","Data":"4ed423139336b82a159517f2298c1a45926f7e6997a830236453c194619d9e5c"} Oct 30 20:27:38.352325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.351917 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9b04324-e2ac-435b-8756-3812700e9af7","Type":"ContainerDied","Data":"b23bf95a50b4927953fb449472b426f4708434cbcc47cd8cf8ac35763f3f3a31"} Oct 30 20:27:38.352325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.351941 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9b04324-e2ac-435b-8756-3812700e9af7","Type":"ContainerDied","Data":"5c0d7597c7e82e7d2f0db173ad6ae4effb8be243d563b2b64aa1155a98aaa495"} Oct 30 20:27:38.358083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.358065 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4675d67e-7a6f-4959-a883-6356e7ac06f0-tmp\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.358141 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.358101 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4675d67e-7a6f-4959-a883-6356e7ac06f0-console-oauth-config\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.358141 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.358117 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btk2j\" (UniqueName: \"kubernetes.io/projected/4675d67e-7a6f-4959-a883-6356e7ac06f0-kube-api-access-btk2j\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.358293 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.358272 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4675d67e-7a6f-4959-a883-6356e7ac06f0-trusted-ca-bundle\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.358357 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.358317 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4675d67e-7a6f-4959-a883-6356e7ac06f0-service-ca\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.358357 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.358348 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4675d67e-7a6f-4959-a883-6356e7ac06f0-console-serving-cert\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.358499 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.358435 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4675d67e-7a6f-4959-a883-6356e7ac06f0-console-config\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.358580 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.358553 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4675d67e-7a6f-4959-a883-6356e7ac06f0-oauth-serving-cert\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.459574 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.459542 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4675d67e-7a6f-4959-a883-6356e7ac06f0-trusted-ca-bundle\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.459789 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.459589 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4675d67e-7a6f-4959-a883-6356e7ac06f0-service-ca\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.459789 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.459711 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4675d67e-7a6f-4959-a883-6356e7ac06f0-console-serving-cert\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.459789 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.459746 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4675d67e-7a6f-4959-a883-6356e7ac06f0-console-config\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.459954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.459812 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4675d67e-7a6f-4959-a883-6356e7ac06f0-oauth-serving-cert\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.459954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.459853 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4675d67e-7a6f-4959-a883-6356e7ac06f0-tmp\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.459954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.459896 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4675d67e-7a6f-4959-a883-6356e7ac06f0-console-oauth-config\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.459954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.459917 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btk2j\" (UniqueName: \"kubernetes.io/projected/4675d67e-7a6f-4959-a883-6356e7ac06f0-kube-api-access-btk2j\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.460191 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.460047 5755 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 20:27:38.460459 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.460416 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4675d67e-7a6f-4959-a883-6356e7ac06f0-tmp\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.460459 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.460444 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4675d67e-7a6f-4959-a883-6356e7ac06f0-service-ca\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.460612 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.460551 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4675d67e-7a6f-4959-a883-6356e7ac06f0-console-config\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.460612 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.460579 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4675d67e-7a6f-4959-a883-6356e7ac06f0-oauth-serving-cert\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.461223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.461202 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4675d67e-7a6f-4959-a883-6356e7ac06f0-trusted-ca-bundle\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.462278 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.462251 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4675d67e-7a6f-4959-a883-6356e7ac06f0-console-serving-cert\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.462426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.462403 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4675d67e-7a6f-4959-a883-6356e7ac06f0-console-oauth-config\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.518019 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.517993 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btk2j\" (UniqueName: \"kubernetes.io/projected/4675d67e-7a6f-4959-a883-6356e7ac06f0-kube-api-access-btk2j\") pod \"console-57db778948-tm9wd\" (UID: \"4675d67e-7a6f-4959-a883-6356e7ac06f0\") " pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.546946 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.546876 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:38.945961 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:38.945894 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57db778948-tm9wd"] Oct 30 20:27:38.948831 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:38.948806 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4675d67e_7a6f_4959_a883_6356e7ac06f0.slice/crio-170e05deced1c29a8ad8f018dc06aa315b0f3daff6380d198a7f57539c7e0b41 WatchSource:0}: Error finding container 170e05deced1c29a8ad8f018dc06aa315b0f3daff6380d198a7f57539c7e0b41: Status 404 returned error can't find the container with id 170e05deced1c29a8ad8f018dc06aa315b0f3daff6380d198a7f57539c7e0b41 Oct 30 20:27:39.259055 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.259029 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:39.356958 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.356921 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57db778948-tm9wd" event={"ID":"4675d67e-7a6f-4959-a883-6356e7ac06f0","Type":"ContainerStarted","Data":"5480471ebbc5a54b5aa85c51395a1fd0e0a98d501f0f13ae63af6b831df1cd1b"} Oct 30 20:27:39.357139 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.356981 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57db778948-tm9wd" event={"ID":"4675d67e-7a6f-4959-a883-6356e7ac06f0","Type":"ContainerStarted","Data":"170e05deced1c29a8ad8f018dc06aa315b0f3daff6380d198a7f57539c7e0b41"} Oct 30 20:27:39.360007 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.359984 5755 generic.go:356] "Generic (PLEG): container finished" podID="f9b04324-e2ac-435b-8756-3812700e9af7" containerID="45619b244dae9537e65d1c12498cac559619b243fb5f12ac0754454a4d154c89" exitCode=0 Oct 30 20:27:39.360117 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.360053 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9b04324-e2ac-435b-8756-3812700e9af7","Type":"ContainerDied","Data":"45619b244dae9537e65d1c12498cac559619b243fb5f12ac0754454a4d154c89"} Oct 30 20:27:39.360117 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.360061 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:27:39.360117 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.360099 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f9b04324-e2ac-435b-8756-3812700e9af7","Type":"ContainerDied","Data":"a5002587a4732072cddd53b1bf39d63e1ad3aaf071fc57b426faf8c77402ec03"} Oct 30 20:27:39.360207 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.360133 5755 scope.go:122] "RemoveContainer" containerID="02959c2b916e35e40644a616fd4bd315a1266f7b98d5f8b77e8bc1d8abbc370b" Oct 30 20:27:39.367790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.367773 5755 scope.go:122] "RemoveContainer" containerID="4e2ea3b15932e039201023f2710e7c35771606421e5173c96d40ce08a46a8a58" Oct 30 20:27:39.367912 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.367887 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-main-db\") pod \"f9b04324-e2ac-435b-8756-3812700e9af7\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " Oct 30 20:27:39.368032 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.367931 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-tls-assets\" (UniqueName: \"kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-tls-assets\") pod \"f9b04324-e2ac-435b-8756-3812700e9af7\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " Oct 30 20:27:39.368032 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.367963 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-kube-api-access-hd89k\" (UniqueName: \"kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-kube-api-access-hd89k\") pod \"f9b04324-e2ac-435b-8756-3812700e9af7\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " Oct 30 20:27:39.368032 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.368004 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy\") pod \"f9b04324-e2ac-435b-8756-3812700e9af7\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " Oct 30 20:27:39.368188 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.368033 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-web\") pod \"f9b04324-e2ac-435b-8756-3812700e9af7\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " Oct 30 20:27:39.368188 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.368068 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-web-config\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-web-config\") pod \"f9b04324-e2ac-435b-8756-3812700e9af7\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " Oct 30 20:27:39.368188 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.368106 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-trusted-ca-bundle\") pod \"f9b04324-e2ac-435b-8756-3812700e9af7\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " Oct 30 20:27:39.368188 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.368139 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-config-out\") pod \"f9b04324-e2ac-435b-8756-3812700e9af7\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " Oct 30 20:27:39.368188 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.368164 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-main-db" pod "f9b04324-e2ac-435b-8756-3812700e9af7" (UID: "f9b04324-e2ac-435b-8756-3812700e9af7"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:39.368471 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.368171 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-cluster-tls-config\") pod \"f9b04324-e2ac-435b-8756-3812700e9af7\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " Oct 30 20:27:39.368471 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.368287 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-metrics-client-ca\") pod \"f9b04324-e2ac-435b-8756-3812700e9af7\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " Oct 30 20:27:39.368471 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.368365 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-main-tls\") pod \"f9b04324-e2ac-435b-8756-3812700e9af7\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " Oct 30 20:27:39.368471 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.368437 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-config-volume\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-config-volume\") pod \"f9b04324-e2ac-435b-8756-3812700e9af7\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " Oct 30 20:27:39.368827 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.368482 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f9b04324-e2ac-435b-8756-3812700e9af7\" (UID: \"f9b04324-e2ac-435b-8756-3812700e9af7\") " Oct 30 20:27:39.368959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.368928 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-trusted-ca-bundle" pod "f9b04324-e2ac-435b-8756-3812700e9af7" (UID: "f9b04324-e2ac-435b-8756-3812700e9af7"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:39.369856 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.369829 5755 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-main-db\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:39.369948 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.369866 5755 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:39.371954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.371923 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-kube-api-access-hd89k" pod "f9b04324-e2ac-435b-8756-3812700e9af7" (UID: "f9b04324-e2ac-435b-8756-3812700e9af7"). InnerVolumeSpecName "kube-api-access-hd89k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:39.372673 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.372646 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-config-out" pod "f9b04324-e2ac-435b-8756-3812700e9af7" (UID: "f9b04324-e2ac-435b-8756-3812700e9af7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:39.372915 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.372893 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-tls-assets" pod "f9b04324-e2ac-435b-8756-3812700e9af7" (UID: "f9b04324-e2ac-435b-8756-3812700e9af7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:39.373299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.373273 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-metrics-client-ca" pod "f9b04324-e2ac-435b-8756-3812700e9af7" (UID: "f9b04324-e2ac-435b-8756-3812700e9af7"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:39.373589 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.373502 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy" pod "f9b04324-e2ac-435b-8756-3812700e9af7" (UID: "f9b04324-e2ac-435b-8756-3812700e9af7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:39.373918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.373876 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-web" pod "f9b04324-e2ac-435b-8756-3812700e9af7" (UID: "f9b04324-e2ac-435b-8756-3812700e9af7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:39.374004 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.373954 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-metric" pod "f9b04324-e2ac-435b-8756-3812700e9af7" (UID: "f9b04324-e2ac-435b-8756-3812700e9af7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:39.374203 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.374182 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-main-tls" pod "f9b04324-e2ac-435b-8756-3812700e9af7" (UID: "f9b04324-e2ac-435b-8756-3812700e9af7"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:39.375474 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.375453 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-config-volume" pod "f9b04324-e2ac-435b-8756-3812700e9af7" (UID: "f9b04324-e2ac-435b-8756-3812700e9af7"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:39.377296 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.377270 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-cluster-tls-config" pod "f9b04324-e2ac-435b-8756-3812700e9af7" (UID: "f9b04324-e2ac-435b-8756-3812700e9af7"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:39.383530 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.383504 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-web-config" pod "f9b04324-e2ac-435b-8756-3812700e9af7" (UID: "f9b04324-e2ac-435b-8756-3812700e9af7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:39.384618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.384514 5755 scope.go:122] "RemoveContainer" containerID="4ed423139336b82a159517f2298c1a45926f7e6997a830236453c194619d9e5c" Oct 30 20:27:39.391369 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.391348 5755 scope.go:122] "RemoveContainer" containerID="45619b244dae9537e65d1c12498cac559619b243fb5f12ac0754454a4d154c89" Oct 30 20:27:39.398063 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.398046 5755 scope.go:122] "RemoveContainer" containerID="b23bf95a50b4927953fb449472b426f4708434cbcc47cd8cf8ac35763f3f3a31" Oct 30 20:27:39.405077 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.405059 5755 scope.go:122] "RemoveContainer" containerID="5c0d7597c7e82e7d2f0db173ad6ae4effb8be243d563b2b64aa1155a98aaa495" Oct 30 20:27:39.412144 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.412126 5755 scope.go:122] "RemoveContainer" containerID="96f6550998f48971ac1ecf6f3bded5dd68338eec40b2f3f7df8c5567f1035af2" Oct 30 20:27:39.418926 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.418910 5755 scope.go:122] "RemoveContainer" containerID="02959c2b916e35e40644a616fd4bd315a1266f7b98d5f8b77e8bc1d8abbc370b" Oct 30 20:27:39.419184 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:39.419159 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02959c2b916e35e40644a616fd4bd315a1266f7b98d5f8b77e8bc1d8abbc370b\": container with ID starting with 02959c2b916e35e40644a616fd4bd315a1266f7b98d5f8b77e8bc1d8abbc370b not found: ID does not exist" containerID="02959c2b916e35e40644a616fd4bd315a1266f7b98d5f8b77e8bc1d8abbc370b" Oct 30 20:27:39.419233 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.419200 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02959c2b916e35e40644a616fd4bd315a1266f7b98d5f8b77e8bc1d8abbc370b"} err="failed to get container status \"02959c2b916e35e40644a616fd4bd315a1266f7b98d5f8b77e8bc1d8abbc370b\": rpc error: code = NotFound desc = could not find container \"02959c2b916e35e40644a616fd4bd315a1266f7b98d5f8b77e8bc1d8abbc370b\": container with ID starting with 02959c2b916e35e40644a616fd4bd315a1266f7b98d5f8b77e8bc1d8abbc370b not found: ID does not exist" Oct 30 20:27:39.419280 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.419235 5755 scope.go:122] "RemoveContainer" containerID="4e2ea3b15932e039201023f2710e7c35771606421e5173c96d40ce08a46a8a58" Oct 30 20:27:39.419508 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:39.419491 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2ea3b15932e039201023f2710e7c35771606421e5173c96d40ce08a46a8a58\": container with ID starting with 4e2ea3b15932e039201023f2710e7c35771606421e5173c96d40ce08a46a8a58 not found: ID does not exist" containerID="4e2ea3b15932e039201023f2710e7c35771606421e5173c96d40ce08a46a8a58" Oct 30 20:27:39.419553 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.419514 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2ea3b15932e039201023f2710e7c35771606421e5173c96d40ce08a46a8a58"} err="failed to get container status \"4e2ea3b15932e039201023f2710e7c35771606421e5173c96d40ce08a46a8a58\": rpc error: code = NotFound desc = could not find container \"4e2ea3b15932e039201023f2710e7c35771606421e5173c96d40ce08a46a8a58\": container with ID starting with 4e2ea3b15932e039201023f2710e7c35771606421e5173c96d40ce08a46a8a58 not found: ID does not exist" Oct 30 20:27:39.419553 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.419529 5755 scope.go:122] "RemoveContainer" containerID="4ed423139336b82a159517f2298c1a45926f7e6997a830236453c194619d9e5c" Oct 30 20:27:39.419771 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:39.419754 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ed423139336b82a159517f2298c1a45926f7e6997a830236453c194619d9e5c\": container with ID starting with 4ed423139336b82a159517f2298c1a45926f7e6997a830236453c194619d9e5c not found: ID does not exist" containerID="4ed423139336b82a159517f2298c1a45926f7e6997a830236453c194619d9e5c" Oct 30 20:27:39.419824 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.419779 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed423139336b82a159517f2298c1a45926f7e6997a830236453c194619d9e5c"} err="failed to get container status \"4ed423139336b82a159517f2298c1a45926f7e6997a830236453c194619d9e5c\": rpc error: code = NotFound desc = could not find container \"4ed423139336b82a159517f2298c1a45926f7e6997a830236453c194619d9e5c\": container with ID starting with 4ed423139336b82a159517f2298c1a45926f7e6997a830236453c194619d9e5c not found: ID does not exist" Oct 30 20:27:39.419824 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.419801 5755 scope.go:122] "RemoveContainer" containerID="45619b244dae9537e65d1c12498cac559619b243fb5f12ac0754454a4d154c89" Oct 30 20:27:39.420039 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:39.420021 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45619b244dae9537e65d1c12498cac559619b243fb5f12ac0754454a4d154c89\": container with ID starting with 45619b244dae9537e65d1c12498cac559619b243fb5f12ac0754454a4d154c89 not found: ID does not exist" containerID="45619b244dae9537e65d1c12498cac559619b243fb5f12ac0754454a4d154c89" Oct 30 20:27:39.420078 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.420044 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45619b244dae9537e65d1c12498cac559619b243fb5f12ac0754454a4d154c89"} err="failed to get container status \"45619b244dae9537e65d1c12498cac559619b243fb5f12ac0754454a4d154c89\": rpc error: code = NotFound desc = could not find container \"45619b244dae9537e65d1c12498cac559619b243fb5f12ac0754454a4d154c89\": container with ID starting with 45619b244dae9537e65d1c12498cac559619b243fb5f12ac0754454a4d154c89 not found: ID does not exist" Oct 30 20:27:39.420078 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.420059 5755 scope.go:122] "RemoveContainer" containerID="b23bf95a50b4927953fb449472b426f4708434cbcc47cd8cf8ac35763f3f3a31" Oct 30 20:27:39.420287 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:39.420275 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b23bf95a50b4927953fb449472b426f4708434cbcc47cd8cf8ac35763f3f3a31\": container with ID starting with b23bf95a50b4927953fb449472b426f4708434cbcc47cd8cf8ac35763f3f3a31 not found: ID does not exist" containerID="b23bf95a50b4927953fb449472b426f4708434cbcc47cd8cf8ac35763f3f3a31" Oct 30 20:27:39.420320 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.420290 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b23bf95a50b4927953fb449472b426f4708434cbcc47cd8cf8ac35763f3f3a31"} err="failed to get container status \"b23bf95a50b4927953fb449472b426f4708434cbcc47cd8cf8ac35763f3f3a31\": rpc error: code = NotFound desc = could not find container \"b23bf95a50b4927953fb449472b426f4708434cbcc47cd8cf8ac35763f3f3a31\": container with ID starting with b23bf95a50b4927953fb449472b426f4708434cbcc47cd8cf8ac35763f3f3a31 not found: ID does not exist" Oct 30 20:27:39.420320 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.420301 5755 scope.go:122] "RemoveContainer" containerID="5c0d7597c7e82e7d2f0db173ad6ae4effb8be243d563b2b64aa1155a98aaa495" Oct 30 20:27:39.420504 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:39.420489 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0d7597c7e82e7d2f0db173ad6ae4effb8be243d563b2b64aa1155a98aaa495\": container with ID starting with 5c0d7597c7e82e7d2f0db173ad6ae4effb8be243d563b2b64aa1155a98aaa495 not found: ID does not exist" containerID="5c0d7597c7e82e7d2f0db173ad6ae4effb8be243d563b2b64aa1155a98aaa495" Oct 30 20:27:39.420552 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.420508 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0d7597c7e82e7d2f0db173ad6ae4effb8be243d563b2b64aa1155a98aaa495"} err="failed to get container status \"5c0d7597c7e82e7d2f0db173ad6ae4effb8be243d563b2b64aa1155a98aaa495\": rpc error: code = NotFound desc = could not find container \"5c0d7597c7e82e7d2f0db173ad6ae4effb8be243d563b2b64aa1155a98aaa495\": container with ID starting with 5c0d7597c7e82e7d2f0db173ad6ae4effb8be243d563b2b64aa1155a98aaa495 not found: ID does not exist" Oct 30 20:27:39.420552 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.420518 5755 scope.go:122] "RemoveContainer" containerID="96f6550998f48971ac1ecf6f3bded5dd68338eec40b2f3f7df8c5567f1035af2" Oct 30 20:27:39.420717 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:39.420701 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f6550998f48971ac1ecf6f3bded5dd68338eec40b2f3f7df8c5567f1035af2\": container with ID starting with 96f6550998f48971ac1ecf6f3bded5dd68338eec40b2f3f7df8c5567f1035af2 not found: ID does not exist" containerID="96f6550998f48971ac1ecf6f3bded5dd68338eec40b2f3f7df8c5567f1035af2" Oct 30 20:27:39.420768 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.420727 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f6550998f48971ac1ecf6f3bded5dd68338eec40b2f3f7df8c5567f1035af2"} err="failed to get container status \"96f6550998f48971ac1ecf6f3bded5dd68338eec40b2f3f7df8c5567f1035af2\": rpc error: code = NotFound desc = could not find container \"96f6550998f48971ac1ecf6f3bded5dd68338eec40b2f3f7df8c5567f1035af2\": container with ID starting with 96f6550998f48971ac1ecf6f3bded5dd68338eec40b2f3f7df8c5567f1035af2 not found: ID does not exist" Oct 30 20:27:39.471206 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.471009 5755 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-config-volume\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:39.471206 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.471043 5755 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:39.471206 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.471059 5755 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-tls-assets\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:39.471206 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.471073 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd89k\" (UniqueName: \"kubernetes.io/projected/f9b04324-e2ac-435b-8756-3812700e9af7-kube-api-access-hd89k\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:39.471206 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.471089 5755 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:39.471206 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.471105 5755 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:39.471206 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.471121 5755 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-web-config\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:39.471206 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.471136 5755 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9b04324-e2ac-435b-8756-3812700e9af7-config-out\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:39.471206 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.471150 5755 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-cluster-tls-config\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:39.471206 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.471166 5755 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9b04324-e2ac-435b-8756-3812700e9af7-metrics-client-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:39.471206 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.471180 5755 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f9b04324-e2ac-435b-8756-3812700e9af7-secret-alertmanager-main-tls\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:39.505793 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.505683 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-57db778948-tm9wd" podStartSLOduration=1.5056638329999998 podStartE2EDuration="1.505663833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:39.504785373 +0000 UTC m=+22.431342304" watchObservedRunningTime="2026-10-30 20:27:39.505663833 +0000 UTC m=+22.432220752" Oct 30 20:27:39.786310 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.786228 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:27:39.792784 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:39.792757 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:27:41.209670 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:41.209634 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" path="/var/lib/kubelet/pods/f9b04324-e2ac-435b-8756-3812700e9af7/volumes" Oct 30 20:27:41.254031 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:41.253989 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:27:43.289979 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.289927 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-77c5f96b9d-ll5kd" podUID="5ea8cfb3-a932-4689-8eda-f25dc44ca94a" containerName="console" containerID="cri-o://353f1b1f4cd3fc6e41ba880f4fa88877efb099ff32a6cf8ab1c293d2800d9786" gracePeriod=15 Oct 30 20:27:43.291658 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.291624 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6cf86b8865-9l5xr" podUID="44293dda-24ae-4aae-aa97-1e262c3186de" containerName="console" containerID="cri-o://f034529fd12dfcd14e0bb6d373ae32876cf03799f83e2fd25d38fafb0ccefd2b" gracePeriod=15 Oct 30 20:27:43.543055 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.542972 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:43.587714 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.587693 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:43.708587 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.708559 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-trusted-ca-bundle\") pod \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " Oct 30 20:27:43.708751 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.708614 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-trusted-ca-bundle\") pod \"44293dda-24ae-4aae-aa97-1e262c3186de\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " Oct 30 20:27:43.708751 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.708650 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-service-ca\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-service-ca\") pod \"44293dda-24ae-4aae-aa97-1e262c3186de\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " Oct 30 20:27:43.708751 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.708672 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44293dda-24ae-4aae-aa97-1e262c3186de-kube-api-access-w4fnf\" (UniqueName: \"kubernetes.io/projected/44293dda-24ae-4aae-aa97-1e262c3186de-kube-api-access-w4fnf\") pod \"44293dda-24ae-4aae-aa97-1e262c3186de\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " Oct 30 20:27:43.708751 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.708706 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-config\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-config\") pod \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " Oct 30 20:27:43.708751 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.708733 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-oauth-serving-cert\") pod \"44293dda-24ae-4aae-aa97-1e262c3186de\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " Oct 30 20:27:43.708993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.708817 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-service-ca\") pod \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " Oct 30 20:27:43.708993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.708903 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-kube-api-access-klngr\" (UniqueName: \"kubernetes.io/projected/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-kube-api-access-klngr\") pod \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " Oct 30 20:27:43.708993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.708965 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44293dda-24ae-4aae-aa97-1e262c3186de-tmp\" (UniqueName: \"kubernetes.io/empty-dir/44293dda-24ae-4aae-aa97-1e262c3186de-tmp\") pod \"44293dda-24ae-4aae-aa97-1e262c3186de\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " Oct 30 20:27:43.709138 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.708999 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-oauth-serving-cert\") pod \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " Oct 30 20:27:43.709138 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.709031 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-console-config\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-console-config\") pod \"44293dda-24ae-4aae-aa97-1e262c3186de\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " Oct 30 20:27:43.709138 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.709075 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-tmp\") pod \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " Oct 30 20:27:43.709138 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.709110 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-serving-cert\") pod \"44293dda-24ae-4aae-aa97-1e262c3186de\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " Oct 30 20:27:43.709322 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.709140 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-serving-cert\") pod \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " Oct 30 20:27:43.709322 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.709173 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-oauth-config\") pod \"44293dda-24ae-4aae-aa97-1e262c3186de\" (UID: \"44293dda-24ae-4aae-aa97-1e262c3186de\") " Oct 30 20:27:43.709322 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.709201 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-oauth-config\") pod \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\" (UID: \"5ea8cfb3-a932-4689-8eda-f25dc44ca94a\") " Oct 30 20:27:43.709838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.709069 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-trusted-ca-bundle" pod "5ea8cfb3-a932-4689-8eda-f25dc44ca94a" (UID: "5ea8cfb3-a932-4689-8eda-f25dc44ca94a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:43.709838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.709797 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-console-config" pod "44293dda-24ae-4aae-aa97-1e262c3186de" (UID: "44293dda-24ae-4aae-aa97-1e262c3186de"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:43.709838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.709159 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-config" pod "5ea8cfb3-a932-4689-8eda-f25dc44ca94a" (UID: "5ea8cfb3-a932-4689-8eda-f25dc44ca94a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:43.709838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.709194 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-trusted-ca-bundle" pod "44293dda-24ae-4aae-aa97-1e262c3186de" (UID: "44293dda-24ae-4aae-aa97-1e262c3186de"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:43.709838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.709197 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-service-ca" pod "5ea8cfb3-a932-4689-8eda-f25dc44ca94a" (UID: "5ea8cfb3-a932-4689-8eda-f25dc44ca94a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:43.709838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.709297 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44293dda-24ae-4aae-aa97-1e262c3186de-tmp" pod "44293dda-24ae-4aae-aa97-1e262c3186de" (UID: "44293dda-24ae-4aae-aa97-1e262c3186de"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:43.710425 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.709379 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-oauth-serving-cert" pod "44293dda-24ae-4aae-aa97-1e262c3186de" (UID: "44293dda-24ae-4aae-aa97-1e262c3186de"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:43.710425 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.709479 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-tmp" pod "5ea8cfb3-a932-4689-8eda-f25dc44ca94a" (UID: "5ea8cfb3-a932-4689-8eda-f25dc44ca94a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:43.710425 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.709564 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-service-ca" pod "44293dda-24ae-4aae-aa97-1e262c3186de" (UID: "44293dda-24ae-4aae-aa97-1e262c3186de"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:43.710425 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.709773 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-oauth-serving-cert" pod "5ea8cfb3-a932-4689-8eda-f25dc44ca94a" (UID: "5ea8cfb3-a932-4689-8eda-f25dc44ca94a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:43.712008 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.711984 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-kube-api-access-klngr" pod "5ea8cfb3-a932-4689-8eda-f25dc44ca94a" (UID: "5ea8cfb3-a932-4689-8eda-f25dc44ca94a"). InnerVolumeSpecName "kube-api-access-klngr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:43.712116 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.712003 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-serving-cert" pod "5ea8cfb3-a932-4689-8eda-f25dc44ca94a" (UID: "5ea8cfb3-a932-4689-8eda-f25dc44ca94a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:43.712116 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.712010 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44293dda-24ae-4aae-aa97-1e262c3186de-kube-api-access-w4fnf" pod "44293dda-24ae-4aae-aa97-1e262c3186de" (UID: "44293dda-24ae-4aae-aa97-1e262c3186de"). InnerVolumeSpecName "kube-api-access-w4fnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:43.712199 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.712117 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-oauth-config" pod "44293dda-24ae-4aae-aa97-1e262c3186de" (UID: "44293dda-24ae-4aae-aa97-1e262c3186de"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:43.712199 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.712190 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-serving-cert" pod "44293dda-24ae-4aae-aa97-1e262c3186de" (UID: "44293dda-24ae-4aae-aa97-1e262c3186de"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:43.712262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.712208 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-oauth-config" pod "5ea8cfb3-a932-4689-8eda-f25dc44ca94a" (UID: "5ea8cfb3-a932-4689-8eda-f25dc44ca94a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:43.810517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.810446 5755 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44293dda-24ae-4aae-aa97-1e262c3186de-tmp\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:43.810517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.810473 5755 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-oauth-serving-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:43.810517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.810484 5755 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-console-config\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:43.810517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.810492 5755 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-tmp\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:43.810517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.810501 5755 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-serving-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:43.810517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.810510 5755 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-serving-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:43.810517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.810518 5755 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44293dda-24ae-4aae-aa97-1e262c3186de-console-oauth-config\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:43.810831 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.810526 5755 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-oauth-config\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:43.810831 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.810535 5755 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-trusted-ca-bundle\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:43.810831 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.810543 5755 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-trusted-ca-bundle\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:43.810831 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.810551 5755 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:43.810831 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.810560 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4fnf\" (UniqueName: \"kubernetes.io/projected/44293dda-24ae-4aae-aa97-1e262c3186de-kube-api-access-w4fnf\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:43.810831 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.810570 5755 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-console-config\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:43.810831 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.810578 5755 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44293dda-24ae-4aae-aa97-1e262c3186de-oauth-serving-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:43.810831 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.810586 5755 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:43.810831 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.810606 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klngr\" (UniqueName: \"kubernetes.io/projected/5ea8cfb3-a932-4689-8eda-f25dc44ca94a-kube-api-access-klngr\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:43.834229 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.834198 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:27:43.834550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.834509 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="prometheus" containerID="cri-o://3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447" gracePeriod=600 Oct 30 20:27:43.834705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.834532 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="kube-rbac-proxy" containerID="cri-o://626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f" gracePeriod=600 Oct 30 20:27:43.834705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.834619 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="config-reloader" containerID="cri-o://7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8" gracePeriod=600 Oct 30 20:27:43.834705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.834606 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="kube-rbac-proxy-web" containerID="cri-o://e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09" gracePeriod=600 Oct 30 20:27:43.834705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.834546 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="thanos-sidecar" containerID="cri-o://4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d" gracePeriod=600 Oct 30 20:27:43.834705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:43.834572 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="kube-rbac-proxy-thanos" containerID="cri-o://149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c" gracePeriod=600 Oct 30 20:27:44.080317 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.080293 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.214242 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214210 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-serving-certs-ca-bundle\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.214242 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214244 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config-out\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.214499 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214276 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-1\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.214499 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214300 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-thanos-prometheus-http-client-file\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.214499 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214331 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.214499 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214363 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-db\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.214499 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214419 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-metrics-client-ca\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.214762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214533 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-kube-api-access-f7g4d\" (UniqueName: \"kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-kube-api-access-f7g4d\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.214762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214591 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-metrics-client-certs\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.214762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214621 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-kubelet-serving-ca-bundle\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.214762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214665 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-kube-rbac-proxy\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.214762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214687 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-1" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:44.214762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214712 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.214762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214722 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-serving-certs-ca-bundle" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:44.214762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214744 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-2\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.215299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214773 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-tls\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.215299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214801 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-tls-assets\" (UniqueName: \"kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-tls-assets\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.215299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214835 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-web-config\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-web-config\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.215299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214863 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.215299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214892 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-trusted-ca-bundle\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.215299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214945 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-0\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.215299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.214979 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-grpc-tls\") pod \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\" (UID: \"0f928cc6-d12f-42d6-82d3-4673ed8a9d07\") " Oct 30 20:27:44.215299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.215060 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-metrics-client-ca" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:44.215299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.215225 5755 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.215299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.215244 5755 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.215299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.215259 5755 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-metrics-client-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.215846 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.215364 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-db" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:44.216989 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.216037 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-2" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:44.218524 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.218491 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-kube-rbac-proxy" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:44.218656 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.218571 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config-out" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:44.218656 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.218614 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-kubelet-serving-ca-bundle" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:44.218954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.218901 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-trusted-ca-bundle" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:44.219665 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.219550 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-thanos-sidecar-tls" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:44.219859 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.219828 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-metrics-client-certs" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:44.219859 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.219847 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-0" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:44.220008 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.219926 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-kube-rbac-proxy-web" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:44.220077 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.220004 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-thanos-prometheus-http-client-file" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:44.220192 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.220167 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-tls" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:44.220443 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.220417 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-kube-api-access-f7g4d" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "kube-api-access-f7g4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:44.220594 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.220571 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:44.221265 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.221245 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-tls-assets" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:44.221326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.221280 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-grpc-tls" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:44.230579 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.230557 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-web-config" pod "0f928cc6-d12f-42d6-82d3-4673ed8a9d07" (UID: "0f928cc6-d12f-42d6-82d3-4673ed8a9d07"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:44.295440 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.295402 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7d5bb744b-ptq76" podUID="a459d678-44f2-4eed-9cb2-f370f1dcc8c1" containerName="console" containerID="cri-o://fd10c28cba2fb11e7d13d38f7182c7764e914f7aaf741ce3c05d0770b3893c59" gracePeriod=15 Oct 30 20:27:44.316171 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316145 5755 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config-out\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.316171 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316173 5755 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.316325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316184 5755 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.316325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316194 5755 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-db\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.316325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316203 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7g4d\" (UniqueName: \"kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-kube-api-access-f7g4d\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.316325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316212 5755 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-metrics-client-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.316325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316222 5755 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.316325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316231 5755 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-kube-rbac-proxy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.316325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316239 5755 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-config\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.316325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316247 5755 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.316325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316256 5755 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.316325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316263 5755 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-tls-assets\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.316325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316271 5755 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-web-config\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.316325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316281 5755 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.316325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316290 5755 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.316325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316298 5755 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.316325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.316306 5755 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f928cc6-d12f-42d6-82d3-4673ed8a9d07-secret-grpc-tls\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.379631 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.379557 5755 generic.go:356] "Generic (PLEG): container finished" podID="44293dda-24ae-4aae-aa97-1e262c3186de" containerID="f034529fd12dfcd14e0bb6d373ae32876cf03799f83e2fd25d38fafb0ccefd2b" exitCode=0 Oct 30 20:27:44.379631 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.379613 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf86b8865-9l5xr" Oct 30 20:27:44.379825 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.379660 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf86b8865-9l5xr" event={"ID":"44293dda-24ae-4aae-aa97-1e262c3186de","Type":"ContainerDied","Data":"f034529fd12dfcd14e0bb6d373ae32876cf03799f83e2fd25d38fafb0ccefd2b"} Oct 30 20:27:44.379825 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.379722 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf86b8865-9l5xr" event={"ID":"44293dda-24ae-4aae-aa97-1e262c3186de","Type":"ContainerDied","Data":"164bce9e38bd01c82a5295a3639118dbe978703f5028948ab9f98eb52616cb77"} Oct 30 20:27:44.379825 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.379744 5755 scope.go:122] "RemoveContainer" containerID="f034529fd12dfcd14e0bb6d373ae32876cf03799f83e2fd25d38fafb0ccefd2b" Oct 30 20:27:44.382477 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.382454 5755 generic.go:356] "Generic (PLEG): container finished" podID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerID="149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c" exitCode=0 Oct 30 20:27:44.382477 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.382472 5755 generic.go:356] "Generic (PLEG): container finished" podID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerID="626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f" exitCode=0 Oct 30 20:27:44.382477 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.382480 5755 generic.go:356] "Generic (PLEG): container finished" podID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerID="e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09" exitCode=0 Oct 30 20:27:44.382631 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.382486 5755 generic.go:356] "Generic (PLEG): container finished" podID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerID="4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d" exitCode=0 Oct 30 20:27:44.382631 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.382493 5755 generic.go:356] "Generic (PLEG): container finished" podID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerID="7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8" exitCode=0 Oct 30 20:27:44.382631 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.382501 5755 generic.go:356] "Generic (PLEG): container finished" podID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerID="3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447" exitCode=0 Oct 30 20:27:44.382631 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.382530 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.382631 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.382580 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f928cc6-d12f-42d6-82d3-4673ed8a9d07","Type":"ContainerDied","Data":"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c"} Oct 30 20:27:44.382786 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.382647 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f928cc6-d12f-42d6-82d3-4673ed8a9d07","Type":"ContainerDied","Data":"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f"} Oct 30 20:27:44.382786 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.382675 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f928cc6-d12f-42d6-82d3-4673ed8a9d07","Type":"ContainerDied","Data":"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09"} Oct 30 20:27:44.382786 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.382699 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f928cc6-d12f-42d6-82d3-4673ed8a9d07","Type":"ContainerDied","Data":"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d"} Oct 30 20:27:44.382786 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.382728 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f928cc6-d12f-42d6-82d3-4673ed8a9d07","Type":"ContainerDied","Data":"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8"} Oct 30 20:27:44.382786 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.382752 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f928cc6-d12f-42d6-82d3-4673ed8a9d07","Type":"ContainerDied","Data":"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447"} Oct 30 20:27:44.382786 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.382775 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f928cc6-d12f-42d6-82d3-4673ed8a9d07","Type":"ContainerDied","Data":"6e618f27663d49d4bd8025e8fee1cc556fa1117ab56538d3db26cb14e57581d1"} Oct 30 20:27:44.383957 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.383934 5755 generic.go:356] "Generic (PLEG): container finished" podID="5ea8cfb3-a932-4689-8eda-f25dc44ca94a" containerID="353f1b1f4cd3fc6e41ba880f4fa88877efb099ff32a6cf8ab1c293d2800d9786" exitCode=0 Oct 30 20:27:44.384053 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.383987 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c5f96b9d-ll5kd" event={"ID":"5ea8cfb3-a932-4689-8eda-f25dc44ca94a","Type":"ContainerDied","Data":"353f1b1f4cd3fc6e41ba880f4fa88877efb099ff32a6cf8ab1c293d2800d9786"} Oct 30 20:27:44.384053 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.383994 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c5f96b9d-ll5kd" Oct 30 20:27:44.384053 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.384018 5755 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"353f1b1f4cd3fc6e41ba880f4fa88877efb099ff32a6cf8ab1c293d2800d9786"} Oct 30 20:27:44.384053 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.384034 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c5f96b9d-ll5kd" event={"ID":"5ea8cfb3-a932-4689-8eda-f25dc44ca94a","Type":"ContainerDied","Data":"4eff3f83181a243c24aabb7a7daeac647728e7bdfb76044e570a31316e826e1b"} Oct 30 20:27:44.384053 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.384050 5755 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"353f1b1f4cd3fc6e41ba880f4fa88877efb099ff32a6cf8ab1c293d2800d9786"} Oct 30 20:27:44.405335 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.405314 5755 scope.go:122] "RemoveContainer" containerID="f034529fd12dfcd14e0bb6d373ae32876cf03799f83e2fd25d38fafb0ccefd2b" Oct 30 20:27:44.405689 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:44.405671 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f034529fd12dfcd14e0bb6d373ae32876cf03799f83e2fd25d38fafb0ccefd2b\": container with ID starting with f034529fd12dfcd14e0bb6d373ae32876cf03799f83e2fd25d38fafb0ccefd2b not found: ID does not exist" containerID="f034529fd12dfcd14e0bb6d373ae32876cf03799f83e2fd25d38fafb0ccefd2b" Oct 30 20:27:44.405762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.405702 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f034529fd12dfcd14e0bb6d373ae32876cf03799f83e2fd25d38fafb0ccefd2b"} err="failed to get container status \"f034529fd12dfcd14e0bb6d373ae32876cf03799f83e2fd25d38fafb0ccefd2b\": rpc error: code = NotFound desc = could not find container \"f034529fd12dfcd14e0bb6d373ae32876cf03799f83e2fd25d38fafb0ccefd2b\": container with ID starting with f034529fd12dfcd14e0bb6d373ae32876cf03799f83e2fd25d38fafb0ccefd2b not found: ID does not exist" Oct 30 20:27:44.405762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.405723 5755 scope.go:122] "RemoveContainer" containerID="149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c" Oct 30 20:27:44.413012 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.412990 5755 scope.go:122] "RemoveContainer" containerID="626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f" Oct 30 20:27:44.421259 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.421245 5755 scope.go:122] "RemoveContainer" containerID="e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09" Oct 30 20:27:44.428750 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.428729 5755 scope.go:122] "RemoveContainer" containerID="4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d" Oct 30 20:27:44.435876 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.435852 5755 scope.go:122] "RemoveContainer" containerID="7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8" Oct 30 20:27:44.444626 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.444082 5755 scope.go:122] "RemoveContainer" containerID="3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447" Oct 30 20:27:44.452172 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.452146 5755 scope.go:122] "RemoveContainer" containerID="ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2" Oct 30 20:27:44.459712 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.459694 5755 scope.go:122] "RemoveContainer" containerID="149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c" Oct 30 20:27:44.460034 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:44.460014 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c\": container with ID starting with 149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c not found: ID does not exist" containerID="149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c" Oct 30 20:27:44.460105 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.460046 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c"} err="failed to get container status \"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c\": rpc error: code = NotFound desc = could not find container \"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c\": container with ID starting with 149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c not found: ID does not exist" Oct 30 20:27:44.460105 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.460070 5755 scope.go:122] "RemoveContainer" containerID="626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f" Oct 30 20:27:44.460403 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:44.460365 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f\": container with ID starting with 626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f not found: ID does not exist" containerID="626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f" Oct 30 20:27:44.460462 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.460412 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f"} err="failed to get container status \"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f\": rpc error: code = NotFound desc = could not find container \"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f\": container with ID starting with 626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f not found: ID does not exist" Oct 30 20:27:44.460462 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.460434 5755 scope.go:122] "RemoveContainer" containerID="e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09" Oct 30 20:27:44.460692 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:44.460675 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09\": container with ID starting with e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09 not found: ID does not exist" containerID="e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09" Oct 30 20:27:44.460733 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.460697 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09"} err="failed to get container status \"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09\": rpc error: code = NotFound desc = could not find container \"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09\": container with ID starting with e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09 not found: ID does not exist" Oct 30 20:27:44.460733 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.460716 5755 scope.go:122] "RemoveContainer" containerID="4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d" Oct 30 20:27:44.460946 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:44.460928 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d\": container with ID starting with 4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d not found: ID does not exist" containerID="4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d" Oct 30 20:27:44.461008 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.460964 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d"} err="failed to get container status \"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d\": rpc error: code = NotFound desc = could not find container \"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d\": container with ID starting with 4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d not found: ID does not exist" Oct 30 20:27:44.461008 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.460984 5755 scope.go:122] "RemoveContainer" containerID="7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8" Oct 30 20:27:44.461195 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:44.461179 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8\": container with ID starting with 7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8 not found: ID does not exist" containerID="7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8" Oct 30 20:27:44.461240 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.461198 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8"} err="failed to get container status \"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8\": rpc error: code = NotFound desc = could not find container \"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8\": container with ID starting with 7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8 not found: ID does not exist" Oct 30 20:27:44.461240 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.461218 5755 scope.go:122] "RemoveContainer" containerID="3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447" Oct 30 20:27:44.461474 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:44.461454 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447\": container with ID starting with 3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447 not found: ID does not exist" containerID="3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447" Oct 30 20:27:44.461584 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.461500 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447"} err="failed to get container status \"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447\": rpc error: code = NotFound desc = could not find container \"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447\": container with ID starting with 3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447 not found: ID does not exist" Oct 30 20:27:44.461584 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.461516 5755 scope.go:122] "RemoveContainer" containerID="ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2" Oct 30 20:27:44.461747 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:44.461733 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2\": container with ID starting with ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2 not found: ID does not exist" containerID="ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2" Oct 30 20:27:44.461783 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.461751 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2"} err="failed to get container status \"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2\": rpc error: code = NotFound desc = could not find container \"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2\": container with ID starting with ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2 not found: ID does not exist" Oct 30 20:27:44.461783 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.461773 5755 scope.go:122] "RemoveContainer" containerID="149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c" Oct 30 20:27:44.461990 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.461974 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c"} err="failed to get container status \"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c\": rpc error: code = NotFound desc = could not find container \"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c\": container with ID starting with 149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c not found: ID does not exist" Oct 30 20:27:44.461990 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.461989 5755 scope.go:122] "RemoveContainer" containerID="626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f" Oct 30 20:27:44.462228 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.462204 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f"} err="failed to get container status \"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f\": rpc error: code = NotFound desc = could not find container \"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f\": container with ID starting with 626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f not found: ID does not exist" Oct 30 20:27:44.462228 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.462228 5755 scope.go:122] "RemoveContainer" containerID="e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09" Oct 30 20:27:44.462576 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.462559 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09"} err="failed to get container status \"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09\": rpc error: code = NotFound desc = could not find container \"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09\": container with ID starting with e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09 not found: ID does not exist" Oct 30 20:27:44.462576 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.462575 5755 scope.go:122] "RemoveContainer" containerID="4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d" Oct 30 20:27:44.462822 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.462806 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d"} err="failed to get container status \"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d\": rpc error: code = NotFound desc = could not find container \"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d\": container with ID starting with 4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d not found: ID does not exist" Oct 30 20:27:44.462863 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.462822 5755 scope.go:122] "RemoveContainer" containerID="7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8" Oct 30 20:27:44.463063 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.463045 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8"} err="failed to get container status \"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8\": rpc error: code = NotFound desc = could not find container \"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8\": container with ID starting with 7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8 not found: ID does not exist" Oct 30 20:27:44.463104 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.463063 5755 scope.go:122] "RemoveContainer" containerID="3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447" Oct 30 20:27:44.463281 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.463267 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447"} err="failed to get container status \"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447\": rpc error: code = NotFound desc = could not find container \"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447\": container with ID starting with 3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447 not found: ID does not exist" Oct 30 20:27:44.463328 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.463281 5755 scope.go:122] "RemoveContainer" containerID="ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2" Oct 30 20:27:44.463560 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.463546 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2"} err="failed to get container status \"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2\": rpc error: code = NotFound desc = could not find container \"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2\": container with ID starting with ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2 not found: ID does not exist" Oct 30 20:27:44.463609 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.463560 5755 scope.go:122] "RemoveContainer" containerID="149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c" Oct 30 20:27:44.463778 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.463758 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c"} err="failed to get container status \"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c\": rpc error: code = NotFound desc = could not find container \"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c\": container with ID starting with 149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c not found: ID does not exist" Oct 30 20:27:44.463823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.463777 5755 scope.go:122] "RemoveContainer" containerID="626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f" Oct 30 20:27:44.463995 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.463980 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f"} err="failed to get container status \"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f\": rpc error: code = NotFound desc = could not find container \"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f\": container with ID starting with 626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f not found: ID does not exist" Oct 30 20:27:44.464036 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.463995 5755 scope.go:122] "RemoveContainer" containerID="e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09" Oct 30 20:27:44.464201 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.464185 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09"} err="failed to get container status \"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09\": rpc error: code = NotFound desc = could not find container \"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09\": container with ID starting with e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09 not found: ID does not exist" Oct 30 20:27:44.464250 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.464200 5755 scope.go:122] "RemoveContainer" containerID="4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d" Oct 30 20:27:44.464449 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.464431 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d"} err="failed to get container status \"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d\": rpc error: code = NotFound desc = could not find container \"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d\": container with ID starting with 4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d not found: ID does not exist" Oct 30 20:27:44.464499 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.464450 5755 scope.go:122] "RemoveContainer" containerID="7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8" Oct 30 20:27:44.464716 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.464700 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8"} err="failed to get container status \"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8\": rpc error: code = NotFound desc = could not find container \"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8\": container with ID starting with 7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8 not found: ID does not exist" Oct 30 20:27:44.464716 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.464715 5755 scope.go:122] "RemoveContainer" containerID="3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447" Oct 30 20:27:44.465083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.465056 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447"} err="failed to get container status \"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447\": rpc error: code = NotFound desc = could not find container \"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447\": container with ID starting with 3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447 not found: ID does not exist" Oct 30 20:27:44.465083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.465082 5755 scope.go:122] "RemoveContainer" containerID="ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2" Oct 30 20:27:44.465346 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.465326 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2"} err="failed to get container status \"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2\": rpc error: code = NotFound desc = could not find container \"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2\": container with ID starting with ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2 not found: ID does not exist" Oct 30 20:27:44.465459 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.465348 5755 scope.go:122] "RemoveContainer" containerID="149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c" Oct 30 20:27:44.465639 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.465619 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c"} err="failed to get container status \"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c\": rpc error: code = NotFound desc = could not find container \"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c\": container with ID starting with 149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c not found: ID does not exist" Oct 30 20:27:44.465711 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.465640 5755 scope.go:122] "RemoveContainer" containerID="626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f" Oct 30 20:27:44.465967 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.465947 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f"} err="failed to get container status \"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f\": rpc error: code = NotFound desc = could not find container \"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f\": container with ID starting with 626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f not found: ID does not exist" Oct 30 20:27:44.465967 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.465966 5755 scope.go:122] "RemoveContainer" containerID="e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09" Oct 30 20:27:44.466201 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.466186 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09"} err="failed to get container status \"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09\": rpc error: code = NotFound desc = could not find container \"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09\": container with ID starting with e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09 not found: ID does not exist" Oct 30 20:27:44.466201 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.466202 5755 scope.go:122] "RemoveContainer" containerID="4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d" Oct 30 20:27:44.466442 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.466424 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d"} err="failed to get container status \"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d\": rpc error: code = NotFound desc = could not find container \"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d\": container with ID starting with 4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d not found: ID does not exist" Oct 30 20:27:44.466442 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.466442 5755 scope.go:122] "RemoveContainer" containerID="7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8" Oct 30 20:27:44.466725 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.466702 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8"} err="failed to get container status \"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8\": rpc error: code = NotFound desc = could not find container \"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8\": container with ID starting with 7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8 not found: ID does not exist" Oct 30 20:27:44.466725 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.466724 5755 scope.go:122] "RemoveContainer" containerID="3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447" Oct 30 20:27:44.466978 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.466959 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447"} err="failed to get container status \"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447\": rpc error: code = NotFound desc = could not find container \"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447\": container with ID starting with 3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447 not found: ID does not exist" Oct 30 20:27:44.467061 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.466981 5755 scope.go:122] "RemoveContainer" containerID="ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2" Oct 30 20:27:44.467252 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.467234 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2"} err="failed to get container status \"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2\": rpc error: code = NotFound desc = could not find container \"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2\": container with ID starting with ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2 not found: ID does not exist" Oct 30 20:27:44.467304 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.467252 5755 scope.go:122] "RemoveContainer" containerID="149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c" Oct 30 20:27:44.467522 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.467504 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c"} err="failed to get container status \"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c\": rpc error: code = NotFound desc = could not find container \"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c\": container with ID starting with 149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c not found: ID does not exist" Oct 30 20:27:44.467522 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.467523 5755 scope.go:122] "RemoveContainer" containerID="626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f" Oct 30 20:27:44.467772 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.467751 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f"} err="failed to get container status \"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f\": rpc error: code = NotFound desc = could not find container \"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f\": container with ID starting with 626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f not found: ID does not exist" Oct 30 20:27:44.467845 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.467780 5755 scope.go:122] "RemoveContainer" containerID="e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09" Oct 30 20:27:44.468034 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.468011 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09"} err="failed to get container status \"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09\": rpc error: code = NotFound desc = could not find container \"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09\": container with ID starting with e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09 not found: ID does not exist" Oct 30 20:27:44.468100 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.468037 5755 scope.go:122] "RemoveContainer" containerID="4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d" Oct 30 20:27:44.468678 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.468653 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d"} err="failed to get container status \"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d\": rpc error: code = NotFound desc = could not find container \"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d\": container with ID starting with 4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d not found: ID does not exist" Oct 30 20:27:44.468678 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.468676 5755 scope.go:122] "RemoveContainer" containerID="7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8" Oct 30 20:27:44.468998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.468975 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8"} err="failed to get container status \"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8\": rpc error: code = NotFound desc = could not find container \"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8\": container with ID starting with 7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8 not found: ID does not exist" Oct 30 20:27:44.469076 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.468999 5755 scope.go:122] "RemoveContainer" containerID="3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447" Oct 30 20:27:44.470527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.469816 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447"} err="failed to get container status \"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447\": rpc error: code = NotFound desc = could not find container \"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447\": container with ID starting with 3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447 not found: ID does not exist" Oct 30 20:27:44.470527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.469832 5755 scope.go:122] "RemoveContainer" containerID="ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2" Oct 30 20:27:44.470527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.470105 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2"} err="failed to get container status \"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2\": rpc error: code = NotFound desc = could not find container \"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2\": container with ID starting with ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2 not found: ID does not exist" Oct 30 20:27:44.470527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.470139 5755 scope.go:122] "RemoveContainer" containerID="149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c" Oct 30 20:27:44.470751 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.470563 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c"} err="failed to get container status \"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c\": rpc error: code = NotFound desc = could not find container \"149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c\": container with ID starting with 149595d59901ce3a316bbb27f20bae0043717ba76c4d306457ac8775e5a0606c not found: ID does not exist" Oct 30 20:27:44.470751 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.470596 5755 scope.go:122] "RemoveContainer" containerID="626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f" Oct 30 20:27:44.470846 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.470833 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f"} err="failed to get container status \"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f\": rpc error: code = NotFound desc = could not find container \"626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f\": container with ID starting with 626f50a0ce435a9b7f12fc9880da3cf8d87f9e2e02abea129f01b0f55b51866f not found: ID does not exist" Oct 30 20:27:44.470896 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.470849 5755 scope.go:122] "RemoveContainer" containerID="e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09" Oct 30 20:27:44.471087 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.471071 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09"} err="failed to get container status \"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09\": rpc error: code = NotFound desc = could not find container \"e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09\": container with ID starting with e63bbf8d27fcf6e84c6c1430c1c85c3d2ca9ea14e141b726eaa9489faeed6b09 not found: ID does not exist" Oct 30 20:27:44.471268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.471088 5755 scope.go:122] "RemoveContainer" containerID="4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d" Oct 30 20:27:44.471473 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.471455 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d"} err="failed to get container status \"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d\": rpc error: code = NotFound desc = could not find container \"4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d\": container with ID starting with 4af85b226ff4b749e75aced69db1df7fbb6d7bfbc3f481e77c3665ac5305497d not found: ID does not exist" Oct 30 20:27:44.471530 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.471473 5755 scope.go:122] "RemoveContainer" containerID="7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8" Oct 30 20:27:44.471646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.471633 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8"} err="failed to get container status \"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8\": rpc error: code = NotFound desc = could not find container \"7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8\": container with ID starting with 7f3e9da984bf10349d92204c4e1a59ff5457efbce7edb2d68e2ca847936ca5d8 not found: ID does not exist" Oct 30 20:27:44.471707 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.471646 5755 scope.go:122] "RemoveContainer" containerID="3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447" Oct 30 20:27:44.471793 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.471780 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447"} err="failed to get container status \"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447\": rpc error: code = NotFound desc = could not find container \"3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447\": container with ID starting with 3041c9948f071868b55f7ce3d33296f6cf8d4614b39918ce221533bb98a8d447 not found: ID does not exist" Oct 30 20:27:44.471793 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.471792 5755 scope.go:122] "RemoveContainer" containerID="ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2" Oct 30 20:27:44.471937 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.471925 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2"} err="failed to get container status \"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2\": rpc error: code = NotFound desc = could not find container \"ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2\": container with ID starting with ad106b784c1d0ffae43de0f2799fd826655c9b3301d2fc0a6ef8d4051294dec2 not found: ID does not exist" Oct 30 20:27:44.498062 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.498036 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6kfsl" Oct 30 20:27:44.545327 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.545305 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77c5f96b9d-ll5kd"] Oct 30 20:27:44.549817 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.549796 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:44.553589 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.553567 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77c5f96b9d-ll5kd"] Oct 30 20:27:44.664551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.664478 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cf86b8865-9l5xr"] Oct 30 20:27:44.669938 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.669915 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cf86b8865-9l5xr"] Oct 30 20:27:44.719865 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.719833 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-oauth-serving-cert\") pod \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " Oct 30 20:27:44.719934 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.719870 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-kube-api-access-xc7c8\" (UniqueName: \"kubernetes.io/projected/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-kube-api-access-xc7c8\") pod \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " Oct 30 20:27:44.719934 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.719897 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-service-ca\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-service-ca\") pod \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " Oct 30 20:27:44.720017 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.719959 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-oauth-config\") pod \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " Oct 30 20:27:44.720017 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.719993 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-serving-cert\") pod \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " Oct 30 20:27:44.720087 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.720027 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-config\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-config\") pod \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " Oct 30 20:27:44.720087 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.720052 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-tmp\") pod \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\" (UID: \"a459d678-44f2-4eed-9cb2-f370f1dcc8c1\") " Oct 30 20:27:44.720336 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.720306 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-oauth-serving-cert" pod "a459d678-44f2-4eed-9cb2-f370f1dcc8c1" (UID: "a459d678-44f2-4eed-9cb2-f370f1dcc8c1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:44.720504 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.720371 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-service-ca" pod "a459d678-44f2-4eed-9cb2-f370f1dcc8c1" (UID: "a459d678-44f2-4eed-9cb2-f370f1dcc8c1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:44.720622 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.720473 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-tmp" pod "a459d678-44f2-4eed-9cb2-f370f1dcc8c1" (UID: "a459d678-44f2-4eed-9cb2-f370f1dcc8c1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:44.720622 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.720582 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-config" pod "a459d678-44f2-4eed-9cb2-f370f1dcc8c1" (UID: "a459d678-44f2-4eed-9cb2-f370f1dcc8c1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:44.722285 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.722264 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-oauth-config" pod "a459d678-44f2-4eed-9cb2-f370f1dcc8c1" (UID: "a459d678-44f2-4eed-9cb2-f370f1dcc8c1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:44.722514 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.722498 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-kube-api-access-xc7c8" pod "a459d678-44f2-4eed-9cb2-f370f1dcc8c1" (UID: "a459d678-44f2-4eed-9cb2-f370f1dcc8c1"). InnerVolumeSpecName "kube-api-access-xc7c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:44.722616 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.722583 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-serving-cert" pod "a459d678-44f2-4eed-9cb2-f370f1dcc8c1" (UID: "a459d678-44f2-4eed-9cb2-f370f1dcc8c1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:44.784002 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.783963 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:27:44.790300 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.790267 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:27:44.816301 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816276 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:27:44.816553 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816541 5755 cpu_manager.go:414] "containerMap: removing container" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="config-reloader" Oct 30 20:27:44.816592 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816555 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="config-reloader" Oct 30 20:27:44.816592 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816568 5755 cpu_manager.go:414] "containerMap: removing container" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="kube-rbac-proxy" Oct 30 20:27:44.816592 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816572 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="kube-rbac-proxy" Oct 30 20:27:44.816592 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816579 5755 cpu_manager.go:414] "containerMap: removing container" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="prom-label-proxy" Oct 30 20:27:44.816592 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816584 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="prom-label-proxy" Oct 30 20:27:44.816592 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816590 5755 cpu_manager.go:414] "containerMap: removing container" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="kube-rbac-proxy-web" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816595 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="kube-rbac-proxy-web" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816603 5755 cpu_manager.go:414] "containerMap: removing container" podUID="a459d678-44f2-4eed-9cb2-f370f1dcc8c1" containerName="console" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816607 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a459d678-44f2-4eed-9cb2-f370f1dcc8c1" containerName="console" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816612 5755 cpu_manager.go:414] "containerMap: removing container" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="init-config-reloader" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816617 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="init-config-reloader" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816626 5755 cpu_manager.go:414] "containerMap: removing container" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="kube-rbac-proxy-thanos" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816631 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="kube-rbac-proxy-thanos" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816639 5755 cpu_manager.go:414] "containerMap: removing container" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="kube-rbac-proxy" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816643 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="kube-rbac-proxy" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816649 5755 cpu_manager.go:414] "containerMap: removing container" podUID="44293dda-24ae-4aae-aa97-1e262c3186de" containerName="console" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816653 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44293dda-24ae-4aae-aa97-1e262c3186de" containerName="console" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816659 5755 cpu_manager.go:414] "containerMap: removing container" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="init-config-reloader" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816663 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="init-config-reloader" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816670 5755 cpu_manager.go:414] "containerMap: removing container" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="kube-rbac-proxy-web" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816674 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="kube-rbac-proxy-web" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816681 5755 cpu_manager.go:414] "containerMap: removing container" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="kube-rbac-proxy-metric" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816685 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="kube-rbac-proxy-metric" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816691 5755 cpu_manager.go:414] "containerMap: removing container" podUID="5ea8cfb3-a932-4689-8eda-f25dc44ca94a" containerName="console" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816698 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea8cfb3-a932-4689-8eda-f25dc44ca94a" containerName="console" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816704 5755 cpu_manager.go:414] "containerMap: removing container" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="thanos-sidecar" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816708 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="thanos-sidecar" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816714 5755 cpu_manager.go:414] "containerMap: removing container" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="alertmanager" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816718 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="alertmanager" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816726 5755 cpu_manager.go:414] "containerMap: removing container" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="config-reloader" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816730 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="config-reloader" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816737 5755 cpu_manager.go:414] "containerMap: removing container" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="prometheus" Oct 30 20:27:44.816765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816741 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="prometheus" Oct 30 20:27:44.817646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816778 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="kube-rbac-proxy-web" Oct 30 20:27:44.817646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816785 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="kube-rbac-proxy-metric" Oct 30 20:27:44.817646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816790 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea8cfb3-a932-4689-8eda-f25dc44ca94a" containerName="console" Oct 30 20:27:44.817646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816795 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="thanos-sidecar" Oct 30 20:27:44.817646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816799 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="alertmanager" Oct 30 20:27:44.817646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816804 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="config-reloader" Oct 30 20:27:44.817646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816808 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="prometheus" Oct 30 20:27:44.817646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816813 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="config-reloader" Oct 30 20:27:44.817646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816817 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="kube-rbac-proxy" Oct 30 20:27:44.817646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816821 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="prom-label-proxy" Oct 30 20:27:44.817646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816825 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="kube-rbac-proxy-web" Oct 30 20:27:44.817646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816829 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="a459d678-44f2-4eed-9cb2-f370f1dcc8c1" containerName="console" Oct 30 20:27:44.817646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816835 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" containerName="kube-rbac-proxy-thanos" Oct 30 20:27:44.817646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816840 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="kube-rbac-proxy" Oct 30 20:27:44.817646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816843 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="44293dda-24ae-4aae-aa97-1e262c3186de" containerName="console" Oct 30 20:27:44.817646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.816847 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b04324-e2ac-435b-8756-3812700e9af7" containerName="init-config-reloader" Oct 30 20:27:44.819332 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.819318 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.821179 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.821161 5755 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-oauth-config\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.821179 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.821181 5755 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-serving-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.821298 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.821191 5755 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-console-config\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.821298 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.821199 5755 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-tmp\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.821298 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.821209 5755 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-oauth-serving-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.821298 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.821217 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc7c8\" (UniqueName: \"kubernetes.io/projected/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-kube-api-access-xc7c8\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.821298 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.821226 5755 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a459d678-44f2-4eed-9cb2-f370f1dcc8c1-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:27:44.874100 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.874066 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 20:27:44.874100 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.874068 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 20:27:44.874324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.874306 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fk349e28vqu0g\"" Oct 30 20:27:44.874425 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.874345 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 20:27:44.874425 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.874354 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 20:27:44.874527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.874488 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 20:27:44.874527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.874497 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 20:27:44.874663 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.874550 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 20:27:44.874663 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.874568 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 20:27:44.874663 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.874594 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d9pbw\"" Oct 30 20:27:44.874663 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.874663 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 20:27:44.875603 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.875589 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 20:27:44.875885 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.875873 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 20:27:44.876019 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.875993 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 20:27:44.876019 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.876006 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 20:27:44.922071 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922002 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922071 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922036 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922071 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922057 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgjpb\" (UniqueName: \"kubernetes.io/projected/c2d68c75-b92b-419b-b408-a0e0799e7f72-kube-api-access-hgjpb\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922342 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922136 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2d68c75-b92b-419b-b408-a0e0799e7f72-config-out\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922342 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922205 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c2d68c75-b92b-419b-b408-a0e0799e7f72-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922342 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922240 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922342 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922335 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922504 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922371 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2d68c75-b92b-419b-b408-a0e0799e7f72-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922504 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922435 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-web-config\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922504 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922462 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922504 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922485 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922636 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922572 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922636 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922598 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-config\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922636 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922613 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922636 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922629 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922768 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922644 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922768 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922717 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922768 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922741 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922768 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922763 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.922894 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.922815 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:44.926069 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.926054 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 20:27:44.926256 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.926237 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 20:27:44.927178 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:44.927163 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:27:45.024122 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024084 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.024122 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024120 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.024375 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024140 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.024375 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024159 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.024375 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024183 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.024375 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024303 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.024375 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024344 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgjpb\" (UniqueName: \"kubernetes.io/projected/c2d68c75-b92b-419b-b408-a0e0799e7f72-kube-api-access-hgjpb\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.024375 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024368 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2d68c75-b92b-419b-b408-a0e0799e7f72-config-out\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.025262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024415 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c2d68c75-b92b-419b-b408-a0e0799e7f72-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.025262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024441 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.025262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024484 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.025262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024518 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2d68c75-b92b-419b-b408-a0e0799e7f72-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.025262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024541 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-web-config\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.025262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024565 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.025262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024591 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.025262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024638 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.025262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024661 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-config\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.025262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024683 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.025262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024714 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.025262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.024738 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.025262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.025089 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.025262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.025230 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.026014 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.025734 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.026937 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.026684 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c2d68c75-b92b-419b-b408-a0e0799e7f72-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.027029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.026938 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.027767 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.027693 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.028056 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.028031 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.028138 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.028077 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-web-config\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.028430 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.028377 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.028504 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.028438 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.028832 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.028805 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c2d68c75-b92b-419b-b408-a0e0799e7f72-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.028915 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.028818 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.029319 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.029294 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.029540 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.029518 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-config\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.029770 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.029749 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2d68c75-b92b-419b-b408-a0e0799e7f72-config-out\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.029841 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.029778 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.029892 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.029834 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.030492 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.030477 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c2d68c75-b92b-419b-b408-a0e0799e7f72-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.030893 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.030861 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2d68c75-b92b-419b-b408-a0e0799e7f72-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.085981 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.085955 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgjpb\" (UniqueName: \"kubernetes.io/projected/c2d68c75-b92b-419b-b408-a0e0799e7f72-kube-api-access-hgjpb\") pod \"prometheus-k8s-0\" (UID: \"c2d68c75-b92b-419b-b408-a0e0799e7f72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.136001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.135974 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:45.209401 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.209363 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f928cc6-d12f-42d6-82d3-4673ed8a9d07" path="/var/lib/kubelet/pods/0f928cc6-d12f-42d6-82d3-4673ed8a9d07/volumes" Oct 30 20:27:45.210103 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.210088 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44293dda-24ae-4aae-aa97-1e262c3186de" path="/var/lib/kubelet/pods/44293dda-24ae-4aae-aa97-1e262c3186de/volumes" Oct 30 20:27:45.210771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.210757 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea8cfb3-a932-4689-8eda-f25dc44ca94a" path="/var/lib/kubelet/pods/5ea8cfb3-a932-4689-8eda-f25dc44ca94a/volumes" Oct 30 20:27:45.388702 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.388661 5755 generic.go:356] "Generic (PLEG): container finished" podID="a459d678-44f2-4eed-9cb2-f370f1dcc8c1" containerID="fd10c28cba2fb11e7d13d38f7182c7764e914f7aaf741ce3c05d0770b3893c59" exitCode=0 Oct 30 20:27:45.389122 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.388721 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5bb744b-ptq76" Oct 30 20:27:45.389122 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.388736 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5bb744b-ptq76" event={"ID":"a459d678-44f2-4eed-9cb2-f370f1dcc8c1","Type":"ContainerDied","Data":"fd10c28cba2fb11e7d13d38f7182c7764e914f7aaf741ce3c05d0770b3893c59"} Oct 30 20:27:45.389122 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.388808 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5bb744b-ptq76" event={"ID":"a459d678-44f2-4eed-9cb2-f370f1dcc8c1","Type":"ContainerDied","Data":"fbe58304c85541f8664e66f2d8e258a24f94270321951fbbc1e91fb4125b3b83"} Oct 30 20:27:45.389122 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.388829 5755 scope.go:122] "RemoveContainer" containerID="fd10c28cba2fb11e7d13d38f7182c7764e914f7aaf741ce3c05d0770b3893c59" Oct 30 20:27:45.396411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.396374 5755 scope.go:122] "RemoveContainer" containerID="fd10c28cba2fb11e7d13d38f7182c7764e914f7aaf741ce3c05d0770b3893c59" Oct 30 20:27:45.396671 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:27:45.396654 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd10c28cba2fb11e7d13d38f7182c7764e914f7aaf741ce3c05d0770b3893c59\": container with ID starting with fd10c28cba2fb11e7d13d38f7182c7764e914f7aaf741ce3c05d0770b3893c59 not found: ID does not exist" containerID="fd10c28cba2fb11e7d13d38f7182c7764e914f7aaf741ce3c05d0770b3893c59" Oct 30 20:27:45.396742 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.396682 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd10c28cba2fb11e7d13d38f7182c7764e914f7aaf741ce3c05d0770b3893c59"} err="failed to get container status \"fd10c28cba2fb11e7d13d38f7182c7764e914f7aaf741ce3c05d0770b3893c59\": rpc error: code = NotFound desc = could not find container \"fd10c28cba2fb11e7d13d38f7182c7764e914f7aaf741ce3c05d0770b3893c59\": container with ID starting with fd10c28cba2fb11e7d13d38f7182c7764e914f7aaf741ce3c05d0770b3893c59 not found: ID does not exist" Oct 30 20:27:45.554500 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.554440 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d5bb744b-ptq76"] Oct 30 20:27:45.559642 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.559615 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d5bb744b-ptq76"] Oct 30 20:27:45.567741 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:45.567716 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:27:45.571168 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:27:45.571137 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d68c75_b92b_419b_b408_a0e0799e7f72.slice/crio-09e07e32227217a1efe82f73394b902388388a8d0933d2c44d925bf53986e73e WatchSource:0}: Error finding container 09e07e32227217a1efe82f73394b902388388a8d0933d2c44d925bf53986e73e: Status 404 returned error can't find the container with id 09e07e32227217a1efe82f73394b902388388a8d0933d2c44d925bf53986e73e Oct 30 20:27:46.400838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:46.400798 5755 generic.go:356] "Generic (PLEG): container finished" podID="c2d68c75-b92b-419b-b408-a0e0799e7f72" containerID="853ab1d3f64adcd98797fa67132497b4bf9ab862728f9e22771f6ed3a3a97e06" exitCode=0 Oct 30 20:27:46.401257 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:46.400888 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2d68c75-b92b-419b-b408-a0e0799e7f72","Type":"ContainerDied","Data":"853ab1d3f64adcd98797fa67132497b4bf9ab862728f9e22771f6ed3a3a97e06"} Oct 30 20:27:46.401257 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:46.401038 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2d68c75-b92b-419b-b408-a0e0799e7f72","Type":"ContainerStarted","Data":"09e07e32227217a1efe82f73394b902388388a8d0933d2c44d925bf53986e73e"} Oct 30 20:27:47.209667 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:47.209633 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a459d678-44f2-4eed-9cb2-f370f1dcc8c1" path="/var/lib/kubelet/pods/a459d678-44f2-4eed-9cb2-f370f1dcc8c1/volumes" Oct 30 20:27:47.411589 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:47.408483 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2d68c75-b92b-419b-b408-a0e0799e7f72","Type":"ContainerStarted","Data":"80c635e24438c2af69dbe35473d7123ff8238fe5edb4a6b0ee2d38728a7ef5ac"} Oct 30 20:27:47.411589 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:47.408638 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2d68c75-b92b-419b-b408-a0e0799e7f72","Type":"ContainerStarted","Data":"8c4efba1dfdf6a68dd0d1a750409175cbd54b5272a055e45bdc91445de0efa1e"} Oct 30 20:27:47.411589 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:47.408704 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2d68c75-b92b-419b-b408-a0e0799e7f72","Type":"ContainerStarted","Data":"463f728cc449e5c66e796f303c799ac417de2207f69d5e16335ffb6d47e3a092"} Oct 30 20:27:47.411589 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:47.408759 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2d68c75-b92b-419b-b408-a0e0799e7f72","Type":"ContainerStarted","Data":"28ab19e46113a11cc1ee3c1b826198591397d8ce0093330061a9b5a119045165"} Oct 30 20:27:47.411589 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:47.408815 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2d68c75-b92b-419b-b408-a0e0799e7f72","Type":"ContainerStarted","Data":"437f8bb6b5074bbb1258fba7f342c78574a3761457021aa2f315bbb65c805357"} Oct 30 20:27:47.411589 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:47.408871 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2d68c75-b92b-419b-b408-a0e0799e7f72","Type":"ContainerStarted","Data":"4b1f7647c34e45ddbc95522cee19803f3f57d0c54795f0645a1fabadcf18361a"} Oct 30 20:27:47.539314 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:47.539263 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.523429303 podStartE2EDuration="3.539245927s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.816624ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:47.538536367 +0000 UTC m=+30.465093308" watchObservedRunningTime="2026-10-30 20:27:47.539245927 +0000 UTC m=+30.465802837" Oct 30 20:27:47.929220 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:47.929137 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6kqkm" Oct 30 20:27:48.547468 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:48.547426 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:48.547849 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:48.547485 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:48.551978 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:48.551958 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:49.419783 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:49.419753 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57db778948-tm9wd" Oct 30 20:27:49.589040 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:49.589003 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-87bf75d78-kp5hd"] Oct 30 20:27:50.136852 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:50.136814 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:27:50.450783 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:27:50.450700 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tw7jg" Oct 30 20:28:11.253664 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:28:11.253620 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:28:14.608529 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.608467 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-87bf75d78-kp5hd" podUID="e0407316-563c-489c-9826-3c1372a6fbff" containerName="console" containerID="cri-o://1dcf8e094efe59f774bb8877908c8b744891c1161d6e126fd40460cf842221da" gracePeriod=15 Oct 30 20:28:14.839855 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.839832 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:28:14.859931 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.859883 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0407316-563c-489c-9826-3c1372a6fbff-kube-api-access-9fkx6\" (UniqueName: \"kubernetes.io/projected/e0407316-563c-489c-9826-3c1372a6fbff-kube-api-access-9fkx6\") pod \"e0407316-563c-489c-9826-3c1372a6fbff\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " Oct 30 20:28:14.860009 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.859934 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-trusted-ca-bundle\") pod \"e0407316-563c-489c-9826-3c1372a6fbff\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " Oct 30 20:28:14.860009 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.859969 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-oauth-config\") pod \"e0407316-563c-489c-9826-3c1372a6fbff\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " Oct 30 20:28:14.860095 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.860074 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-oauth-serving-cert\") pod \"e0407316-563c-489c-9826-3c1372a6fbff\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " Oct 30 20:28:14.860138 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.860111 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-serving-cert\") pod \"e0407316-563c-489c-9826-3c1372a6fbff\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " Oct 30 20:28:14.860171 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.860152 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0407316-563c-489c-9826-3c1372a6fbff-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0407316-563c-489c-9826-3c1372a6fbff-tmp\") pod \"e0407316-563c-489c-9826-3c1372a6fbff\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " Oct 30 20:28:14.860221 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.860185 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-console-config\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-console-config\") pod \"e0407316-563c-489c-9826-3c1372a6fbff\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " Oct 30 20:28:14.860221 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.860210 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-service-ca\") pod \"e0407316-563c-489c-9826-3c1372a6fbff\" (UID: \"e0407316-563c-489c-9826-3c1372a6fbff\") " Oct 30 20:28:14.860516 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.860377 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-trusted-ca-bundle" pod "e0407316-563c-489c-9826-3c1372a6fbff" (UID: "e0407316-563c-489c-9826-3c1372a6fbff"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:14.860516 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.860500 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0407316-563c-489c-9826-3c1372a6fbff-tmp" pod "e0407316-563c-489c-9826-3c1372a6fbff" (UID: "e0407316-563c-489c-9826-3c1372a6fbff"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:14.860699 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.860524 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-oauth-serving-cert" pod "e0407316-563c-489c-9826-3c1372a6fbff" (UID: "e0407316-563c-489c-9826-3c1372a6fbff"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:14.860757 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.860726 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-service-ca" pod "e0407316-563c-489c-9826-3c1372a6fbff" (UID: "e0407316-563c-489c-9826-3c1372a6fbff"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:14.861029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.860958 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-console-config" pod "e0407316-563c-489c-9826-3c1372a6fbff" (UID: "e0407316-563c-489c-9826-3c1372a6fbff"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:14.862280 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.862259 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-oauth-config" pod "e0407316-563c-489c-9826-3c1372a6fbff" (UID: "e0407316-563c-489c-9826-3c1372a6fbff"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:28:14.862280 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.862269 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0407316-563c-489c-9826-3c1372a6fbff-kube-api-access-9fkx6" pod "e0407316-563c-489c-9826-3c1372a6fbff" (UID: "e0407316-563c-489c-9826-3c1372a6fbff"). InnerVolumeSpecName "kube-api-access-9fkx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:14.862447 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.862360 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-serving-cert" pod "e0407316-563c-489c-9826-3c1372a6fbff" (UID: "e0407316-563c-489c-9826-3c1372a6fbff"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:28:14.961779 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.961746 5755 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0407316-563c-489c-9826-3c1372a6fbff-tmp\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:14.961779 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.961780 5755 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-console-config\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:14.961998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.961793 5755 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:14.961998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.961806 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fkx6\" (UniqueName: \"kubernetes.io/projected/e0407316-563c-489c-9826-3c1372a6fbff-kube-api-access-9fkx6\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:14.961998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.961819 5755 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-trusted-ca-bundle\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:14.961998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.961832 5755 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-oauth-config\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:14.961998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.961844 5755 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0407316-563c-489c-9826-3c1372a6fbff-oauth-serving-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:14.961998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:14.961855 5755 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0407316-563c-489c-9826-3c1372a6fbff-console-serving-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:28:15.509062 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:15.508971 5755 generic.go:356] "Generic (PLEG): container finished" podID="e0407316-563c-489c-9826-3c1372a6fbff" containerID="1dcf8e094efe59f774bb8877908c8b744891c1161d6e126fd40460cf842221da" exitCode=0 Oct 30 20:28:15.509062 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:15.509033 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87bf75d78-kp5hd" Oct 30 20:28:15.509062 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:15.509050 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87bf75d78-kp5hd" event={"ID":"e0407316-563c-489c-9826-3c1372a6fbff","Type":"ContainerDied","Data":"1dcf8e094efe59f774bb8877908c8b744891c1161d6e126fd40460cf842221da"} Oct 30 20:28:15.509293 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:15.509125 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87bf75d78-kp5hd" event={"ID":"e0407316-563c-489c-9826-3c1372a6fbff","Type":"ContainerDied","Data":"9c7aa12704a9e14c288dd5a706e5ef45b8fe03363814b5a5f135fe87662138dd"} Oct 30 20:28:15.509293 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:15.509145 5755 scope.go:122] "RemoveContainer" containerID="1dcf8e094efe59f774bb8877908c8b744891c1161d6e126fd40460cf842221da" Oct 30 20:28:15.517007 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:15.516992 5755 scope.go:122] "RemoveContainer" containerID="1dcf8e094efe59f774bb8877908c8b744891c1161d6e126fd40460cf842221da" Oct 30 20:28:15.517291 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:28:15.517276 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dcf8e094efe59f774bb8877908c8b744891c1161d6e126fd40460cf842221da\": container with ID starting with 1dcf8e094efe59f774bb8877908c8b744891c1161d6e126fd40460cf842221da not found: ID does not exist" containerID="1dcf8e094efe59f774bb8877908c8b744891c1161d6e126fd40460cf842221da" Oct 30 20:28:15.517337 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:15.517299 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dcf8e094efe59f774bb8877908c8b744891c1161d6e126fd40460cf842221da"} err="failed to get container status \"1dcf8e094efe59f774bb8877908c8b744891c1161d6e126fd40460cf842221da\": rpc error: code = NotFound desc = could not find container \"1dcf8e094efe59f774bb8877908c8b744891c1161d6e126fd40460cf842221da\": container with ID starting with 1dcf8e094efe59f774bb8877908c8b744891c1161d6e126fd40460cf842221da not found: ID does not exist" Oct 30 20:28:15.674721 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:15.674688 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-87bf75d78-kp5hd"] Oct 30 20:28:15.724648 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:15.724618 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-87bf75d78-kp5hd"] Oct 30 20:28:17.208404 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:17.208366 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0407316-563c-489c-9826-3c1372a6fbff" path="/var/lib/kubelet/pods/e0407316-563c-489c-9826-3c1372a6fbff/volumes" Oct 30 20:28:27.509571 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:27.509535 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:28:27.525250 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:27.525226 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:28:27.571942 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:28:27.571916 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:28:41.253542 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:28:41.253494 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:29:11.254002 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:29:11.253913 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:29:41.252664 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:29:41.252627 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:30:07.341939 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.341907 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk"] Oct 30 20:30:07.342354 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.342172 5755 cpu_manager.go:414] "containerMap: removing container" podUID="e0407316-563c-489c-9826-3c1372a6fbff" containerName="console" Oct 30 20:30:07.342354 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.342182 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0407316-563c-489c-9826-3c1372a6fbff" containerName="console" Oct 30 20:30:07.342354 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.342235 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0407316-563c-489c-9826-3c1372a6fbff" containerName="console" Oct 30 20:30:07.345404 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.345365 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" Oct 30 20:30:07.416546 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.416521 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmf89\" (UniqueName: \"kubernetes.io/projected/940b9fdc-1f1e-44f4-b718-517956fa735e-kube-api-access-gmf89\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk\" (UID: \"940b9fdc-1f1e-44f4-b718-517956fa735e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" Oct 30 20:30:07.416669 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.416551 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/940b9fdc-1f1e-44f4-b718-517956fa735e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk\" (UID: \"940b9fdc-1f1e-44f4-b718-517956fa735e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" Oct 30 20:30:07.416710 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.416691 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/940b9fdc-1f1e-44f4-b718-517956fa735e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk\" (UID: \"940b9fdc-1f1e-44f4-b718-517956fa735e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" Oct 30 20:30:07.448314 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.448296 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 20:30:07.448419 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.448310 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-lcknn\"" Oct 30 20:30:07.448419 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.448325 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 20:30:07.448920 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.448905 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk"] Oct 30 20:30:07.517505 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.517480 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/940b9fdc-1f1e-44f4-b718-517956fa735e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk\" (UID: \"940b9fdc-1f1e-44f4-b718-517956fa735e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" Oct 30 20:30:07.517608 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.517514 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmf89\" (UniqueName: \"kubernetes.io/projected/940b9fdc-1f1e-44f4-b718-517956fa735e-kube-api-access-gmf89\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk\" (UID: \"940b9fdc-1f1e-44f4-b718-517956fa735e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" Oct 30 20:30:07.517608 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.517538 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/940b9fdc-1f1e-44f4-b718-517956fa735e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk\" (UID: \"940b9fdc-1f1e-44f4-b718-517956fa735e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" Oct 30 20:30:07.517911 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.517889 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/940b9fdc-1f1e-44f4-b718-517956fa735e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk\" (UID: \"940b9fdc-1f1e-44f4-b718-517956fa735e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" Oct 30 20:30:07.517947 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.517899 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/940b9fdc-1f1e-44f4-b718-517956fa735e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk\" (UID: \"940b9fdc-1f1e-44f4-b718-517956fa735e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" Oct 30 20:30:07.575516 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.575495 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmf89\" (UniqueName: \"kubernetes.io/projected/940b9fdc-1f1e-44f4-b718-517956fa735e-kube-api-access-gmf89\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk\" (UID: \"940b9fdc-1f1e-44f4-b718-517956fa735e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" Oct 30 20:30:07.660887 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:07.660826 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" Oct 30 20:30:08.028994 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.028964 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk"] Oct 30 20:30:08.207240 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.207206 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb"] Oct 30 20:30:08.211685 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.211661 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" Oct 30 20:30:08.265374 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.265349 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-lzv6x\"" Oct 30 20:30:08.268508 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.268485 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb"] Oct 30 20:30:08.312641 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.312582 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84"] Oct 30 20:30:08.315725 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.315708 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" Oct 30 20:30:08.329837 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.329813 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg7nv\" (UniqueName: \"kubernetes.io/projected/4b1268c9-2572-4d90-8f9e-551e7747b302-kube-api-access-rg7nv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb\" (UID: \"4b1268c9-2572-4d90-8f9e-551e7747b302\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" Oct 30 20:30:08.329915 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.329857 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b1268c9-2572-4d90-8f9e-551e7747b302-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb\" (UID: \"4b1268c9-2572-4d90-8f9e-551e7747b302\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" Oct 30 20:30:08.330054 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.330035 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b1268c9-2572-4d90-8f9e-551e7747b302-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb\" (UID: \"4b1268c9-2572-4d90-8f9e-551e7747b302\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" Oct 30 20:30:08.369358 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.369333 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-mfdfx\"" Oct 30 20:30:08.379061 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.379039 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84"] Oct 30 20:30:08.411293 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.411265 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958"] Oct 30 20:30:08.414507 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.414489 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" Oct 30 20:30:08.431433 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.431409 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b1268c9-2572-4d90-8f9e-551e7747b302-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb\" (UID: \"4b1268c9-2572-4d90-8f9e-551e7747b302\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" Oct 30 20:30:08.431536 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.431443 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dbcm\" (UniqueName: \"kubernetes.io/projected/240d6e3e-81fe-4415-b581-a7439d0adab0-kube-api-access-2dbcm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84\" (UID: \"240d6e3e-81fe-4415-b581-a7439d0adab0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" Oct 30 20:30:08.431536 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.431488 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg7nv\" (UniqueName: \"kubernetes.io/projected/4b1268c9-2572-4d90-8f9e-551e7747b302-kube-api-access-rg7nv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb\" (UID: \"4b1268c9-2572-4d90-8f9e-551e7747b302\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" Oct 30 20:30:08.431536 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.431514 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b1268c9-2572-4d90-8f9e-551e7747b302-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb\" (UID: \"4b1268c9-2572-4d90-8f9e-551e7747b302\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" Oct 30 20:30:08.431536 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.431530 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/240d6e3e-81fe-4415-b581-a7439d0adab0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84\" (UID: \"240d6e3e-81fe-4415-b581-a7439d0adab0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" Oct 30 20:30:08.431764 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.431563 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/240d6e3e-81fe-4415-b581-a7439d0adab0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84\" (UID: \"240d6e3e-81fe-4415-b581-a7439d0adab0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" Oct 30 20:30:08.431832 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.431785 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b1268c9-2572-4d90-8f9e-551e7747b302-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb\" (UID: \"4b1268c9-2572-4d90-8f9e-551e7747b302\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" Oct 30 20:30:08.431832 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.431822 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b1268c9-2572-4d90-8f9e-551e7747b302-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb\" (UID: \"4b1268c9-2572-4d90-8f9e-551e7747b302\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" Oct 30 20:30:08.469441 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.469413 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-wswwx\"" Oct 30 20:30:08.471980 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.471953 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958"] Oct 30 20:30:08.490008 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.489979 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg7nv\" (UniqueName: \"kubernetes.io/projected/4b1268c9-2572-4d90-8f9e-551e7747b302-kube-api-access-rg7nv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb\" (UID: \"4b1268c9-2572-4d90-8f9e-551e7747b302\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" Oct 30 20:30:08.512197 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.512177 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w"] Oct 30 20:30:08.515441 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.515425 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" Oct 30 20:30:08.527249 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.527231 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" Oct 30 20:30:08.532065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.532048 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dbcm\" (UniqueName: \"kubernetes.io/projected/240d6e3e-81fe-4415-b581-a7439d0adab0-kube-api-access-2dbcm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84\" (UID: \"240d6e3e-81fe-4415-b581-a7439d0adab0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" Oct 30 20:30:08.532130 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.532079 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a94074a3-21bf-48f3-9c01-a7ed53667bd0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958\" (UID: \"a94074a3-21bf-48f3-9c01-a7ed53667bd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" Oct 30 20:30:08.532130 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.532100 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7bq9\" (UniqueName: \"kubernetes.io/projected/a94074a3-21bf-48f3-9c01-a7ed53667bd0-kube-api-access-r7bq9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958\" (UID: \"a94074a3-21bf-48f3-9c01-a7ed53667bd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" Oct 30 20:30:08.532232 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.532216 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a94074a3-21bf-48f3-9c01-a7ed53667bd0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958\" (UID: \"a94074a3-21bf-48f3-9c01-a7ed53667bd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" Oct 30 20:30:08.532271 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.532256 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/240d6e3e-81fe-4415-b581-a7439d0adab0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84\" (UID: \"240d6e3e-81fe-4415-b581-a7439d0adab0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" Oct 30 20:30:08.532324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.532287 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/240d6e3e-81fe-4415-b581-a7439d0adab0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84\" (UID: \"240d6e3e-81fe-4415-b581-a7439d0adab0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" Oct 30 20:30:08.532616 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.532600 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/240d6e3e-81fe-4415-b581-a7439d0adab0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84\" (UID: \"240d6e3e-81fe-4415-b581-a7439d0adab0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" Oct 30 20:30:08.532684 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.532620 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/240d6e3e-81fe-4415-b581-a7439d0adab0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84\" (UID: \"240d6e3e-81fe-4415-b581-a7439d0adab0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" Oct 30 20:30:08.570767 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.570682 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-2nd48\"" Oct 30 20:30:08.577017 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.576428 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w"] Oct 30 20:30:08.590193 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.590160 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dbcm\" (UniqueName: \"kubernetes.io/projected/240d6e3e-81fe-4415-b581-a7439d0adab0-kube-api-access-2dbcm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84\" (UID: \"240d6e3e-81fe-4415-b581-a7439d0adab0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" Oct 30 20:30:08.630873 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.630847 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" Oct 30 20:30:08.633175 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.633156 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a94074a3-21bf-48f3-9c01-a7ed53667bd0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958\" (UID: \"a94074a3-21bf-48f3-9c01-a7ed53667bd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" Oct 30 20:30:08.633262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.633191 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7bq9\" (UniqueName: \"kubernetes.io/projected/a94074a3-21bf-48f3-9c01-a7ed53667bd0-kube-api-access-r7bq9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958\" (UID: \"a94074a3-21bf-48f3-9c01-a7ed53667bd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" Oct 30 20:30:08.633262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.633223 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkq7f\" (UniqueName: \"kubernetes.io/projected/c094def6-82c7-4ac5-b784-ea4bdeb61f67-kube-api-access-wkq7f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w\" (UID: \"c094def6-82c7-4ac5-b784-ea4bdeb61f67\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" Oct 30 20:30:08.633332 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.633291 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a94074a3-21bf-48f3-9c01-a7ed53667bd0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958\" (UID: \"a94074a3-21bf-48f3-9c01-a7ed53667bd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" Oct 30 20:30:08.633411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.633366 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c094def6-82c7-4ac5-b784-ea4bdeb61f67-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w\" (UID: \"c094def6-82c7-4ac5-b784-ea4bdeb61f67\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" Oct 30 20:30:08.633473 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.633431 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c094def6-82c7-4ac5-b784-ea4bdeb61f67-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w\" (UID: \"c094def6-82c7-4ac5-b784-ea4bdeb61f67\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" Oct 30 20:30:08.633571 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.633555 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a94074a3-21bf-48f3-9c01-a7ed53667bd0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958\" (UID: \"a94074a3-21bf-48f3-9c01-a7ed53667bd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" Oct 30 20:30:08.633621 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.633604 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a94074a3-21bf-48f3-9c01-a7ed53667bd0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958\" (UID: \"a94074a3-21bf-48f3-9c01-a7ed53667bd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" Oct 30 20:30:08.692917 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.692885 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7bq9\" (UniqueName: \"kubernetes.io/projected/a94074a3-21bf-48f3-9c01-a7ed53667bd0-kube-api-access-r7bq9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958\" (UID: \"a94074a3-21bf-48f3-9c01-a7ed53667bd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" Oct 30 20:30:08.729757 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.729730 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" Oct 30 20:30:08.734914 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.734890 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkq7f\" (UniqueName: \"kubernetes.io/projected/c094def6-82c7-4ac5-b784-ea4bdeb61f67-kube-api-access-wkq7f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w\" (UID: \"c094def6-82c7-4ac5-b784-ea4bdeb61f67\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" Oct 30 20:30:08.735013 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.734995 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c094def6-82c7-4ac5-b784-ea4bdeb61f67-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w\" (UID: \"c094def6-82c7-4ac5-b784-ea4bdeb61f67\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" Oct 30 20:30:08.735061 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.735045 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c094def6-82c7-4ac5-b784-ea4bdeb61f67-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w\" (UID: \"c094def6-82c7-4ac5-b784-ea4bdeb61f67\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" Oct 30 20:30:08.735349 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.735333 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c094def6-82c7-4ac5-b784-ea4bdeb61f67-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w\" (UID: \"c094def6-82c7-4ac5-b784-ea4bdeb61f67\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" Oct 30 20:30:08.735446 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.735427 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c094def6-82c7-4ac5-b784-ea4bdeb61f67-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w\" (UID: \"c094def6-82c7-4ac5-b784-ea4bdeb61f67\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" Oct 30 20:30:08.793128 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.793101 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkq7f\" (UniqueName: \"kubernetes.io/projected/c094def6-82c7-4ac5-b784-ea4bdeb61f67-kube-api-access-wkq7f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w\" (UID: \"c094def6-82c7-4ac5-b784-ea4bdeb61f67\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" Oct 30 20:30:08.831010 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.830941 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" Oct 30 20:30:08.889976 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.889947 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" event={"ID":"940b9fdc-1f1e-44f4-b718-517956fa735e","Type":"ContainerStarted","Data":"49a24dfd26250f58442ed7791a7b751aab5e3a223174a6fe88a2d8dae0235b68"} Oct 30 20:30:08.895611 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:08.895588 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb"] Oct 30 20:30:08.900222 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:30:08.900193 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1268c9_2572_4d90_8f9e_551e7747b302.slice/crio-8537b20df13004ac90c7fb1ef05ecdd1d730172ff5ecda6dc454fef56c73488f WatchSource:0}: Error finding container 8537b20df13004ac90c7fb1ef05ecdd1d730172ff5ecda6dc454fef56c73488f: Status 404 returned error can't find the container with id 8537b20df13004ac90c7fb1ef05ecdd1d730172ff5ecda6dc454fef56c73488f Oct 30 20:30:09.002683 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:09.002658 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84"] Oct 30 20:30:09.006079 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:30:09.006043 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240d6e3e_81fe_4415_b581_a7439d0adab0.slice/crio-17228da55502689c0fbc828bef44b2e3b0e3564a691b7f1bf8219b67ba3d8157 WatchSource:0}: Error finding container 17228da55502689c0fbc828bef44b2e3b0e3564a691b7f1bf8219b67ba3d8157: Status 404 returned error can't find the container with id 17228da55502689c0fbc828bef44b2e3b0e3564a691b7f1bf8219b67ba3d8157 Oct 30 20:30:09.097367 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:09.097243 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958"] Oct 30 20:30:09.101696 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:30:09.101665 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94074a3_21bf_48f3_9c01_a7ed53667bd0.slice/crio-b5fc75e8c29834da1ba6b1b63ec992b1d1fd8b7adeea8171928e756c5281f5d8 WatchSource:0}: Error finding container b5fc75e8c29834da1ba6b1b63ec992b1d1fd8b7adeea8171928e756c5281f5d8: Status 404 returned error can't find the container with id b5fc75e8c29834da1ba6b1b63ec992b1d1fd8b7adeea8171928e756c5281f5d8 Oct 30 20:30:09.199306 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:09.199273 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w"] Oct 30 20:30:09.897095 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:09.897050 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" event={"ID":"c094def6-82c7-4ac5-b784-ea4bdeb61f67","Type":"ContainerStarted","Data":"8a419a569b8b563efde82f79de741b15809807795ce4ad293ed6441be671571e"} Oct 30 20:30:09.898769 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:09.898734 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" event={"ID":"4b1268c9-2572-4d90-8f9e-551e7747b302","Type":"ContainerStarted","Data":"8537b20df13004ac90c7fb1ef05ecdd1d730172ff5ecda6dc454fef56c73488f"} Oct 30 20:30:09.902049 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:09.902020 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" event={"ID":"a94074a3-21bf-48f3-9c01-a7ed53667bd0","Type":"ContainerStarted","Data":"b5fc75e8c29834da1ba6b1b63ec992b1d1fd8b7adeea8171928e756c5281f5d8"} Oct 30 20:30:09.903260 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:09.903232 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" event={"ID":"240d6e3e-81fe-4415-b581-a7439d0adab0","Type":"ContainerStarted","Data":"17228da55502689c0fbc828bef44b2e3b0e3564a691b7f1bf8219b67ba3d8157"} Oct 30 20:30:10.665729 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:10.665692 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg"] Oct 30 20:30:10.669655 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:10.669610 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" Oct 30 20:30:10.773082 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:10.773048 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg"] Oct 30 20:30:10.773272 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:10.773213 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-jk2md\"" Oct 30 20:30:10.852415 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:10.852362 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg\" (UID: \"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" Oct 30 20:30:10.852644 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:10.852456 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt6wp\" (UniqueName: \"kubernetes.io/projected/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-kube-api-access-rt6wp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg\" (UID: \"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" Oct 30 20:30:10.852644 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:10.852515 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg\" (UID: \"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" Oct 30 20:30:10.954027 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:10.953931 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg\" (UID: \"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" Oct 30 20:30:10.954027 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:10.954019 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg\" (UID: \"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" Oct 30 20:30:10.954590 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:10.954059 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt6wp\" (UniqueName: \"kubernetes.io/projected/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-kube-api-access-rt6wp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg\" (UID: \"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" Oct 30 20:30:10.955006 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:10.954984 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg\" (UID: \"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" Oct 30 20:30:10.955352 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:10.955314 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg\" (UID: \"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" Oct 30 20:30:11.022874 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:11.022810 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt6wp\" (UniqueName: \"kubernetes.io/projected/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-kube-api-access-rt6wp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg\" (UID: \"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" Oct 30 20:30:11.270951 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:30:11.270873 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:30:11.290924 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:11.290891 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" Oct 30 20:30:11.690950 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:11.690873 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg"] Oct 30 20:30:11.699535 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:30:11.699504 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80dc99f7_a894_4dd2_86c3_4a6e7d4ab99e.slice/crio-d8b38aa00e3041b5818a019420d45dfbd7c0ef0ababb02097e957dc0374ccf4d WatchSource:0}: Error finding container d8b38aa00e3041b5818a019420d45dfbd7c0ef0ababb02097e957dc0374ccf4d: Status 404 returned error can't find the container with id d8b38aa00e3041b5818a019420d45dfbd7c0ef0ababb02097e957dc0374ccf4d Oct 30 20:30:11.923552 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:11.923508 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" event={"ID":"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e","Type":"ContainerStarted","Data":"d8b38aa00e3041b5818a019420d45dfbd7c0ef0ababb02097e957dc0374ccf4d"} Oct 30 20:30:20.980288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:20.980251 5755 generic.go:356] "Generic (PLEG): container finished" podID="940b9fdc-1f1e-44f4-b718-517956fa735e" containerID="23651a0aa16da3de6a32e4e5f217338a6c2ac586befb62e7ca1be22119fab113" exitCode=0 Oct 30 20:30:20.980847 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:20.980345 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" event={"ID":"940b9fdc-1f1e-44f4-b718-517956fa735e","Type":"ContainerDied","Data":"23651a0aa16da3de6a32e4e5f217338a6c2ac586befb62e7ca1be22119fab113"} Oct 30 20:30:20.982413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:20.982308 5755 generic.go:356] "Generic (PLEG): container finished" podID="80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e" containerID="c5eceffe9c668655c28b5a6b4e5f8390fe21468fc6a620ffcc0d8a7663b852ca" exitCode=0 Oct 30 20:30:20.982413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:20.982349 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" event={"ID":"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e","Type":"ContainerDied","Data":"c5eceffe9c668655c28b5a6b4e5f8390fe21468fc6a620ffcc0d8a7663b852ca"} Oct 30 20:30:21.987258 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:21.987226 5755 generic.go:356] "Generic (PLEG): container finished" podID="4b1268c9-2572-4d90-8f9e-551e7747b302" containerID="672743a2b0e2922aed29244090c80d6484043a01ce45ac5cbeff10662aa8512b" exitCode=0 Oct 30 20:30:21.987701 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:21.987332 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" event={"ID":"4b1268c9-2572-4d90-8f9e-551e7747b302","Type":"ContainerDied","Data":"672743a2b0e2922aed29244090c80d6484043a01ce45ac5cbeff10662aa8512b"} Oct 30 20:30:21.989476 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:21.989452 5755 generic.go:356] "Generic (PLEG): container finished" podID="240d6e3e-81fe-4415-b581-a7439d0adab0" containerID="07b10df4cc5fe5415c6f922f973a0aaf81809cf4684dc3a6349ca80b6bbcc1ad" exitCode=0 Oct 30 20:30:21.989576 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:21.989514 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" event={"ID":"240d6e3e-81fe-4415-b581-a7439d0adab0","Type":"ContainerDied","Data":"07b10df4cc5fe5415c6f922f973a0aaf81809cf4684dc3a6349ca80b6bbcc1ad"} Oct 30 20:30:21.991086 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:21.991060 5755 generic.go:356] "Generic (PLEG): container finished" podID="c094def6-82c7-4ac5-b784-ea4bdeb61f67" containerID="691b34bec4be7b788037a523724a68cd9d07f2a5e3cf4752fd744dcdcc82f313" exitCode=0 Oct 30 20:30:21.991194 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:21.991085 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" event={"ID":"c094def6-82c7-4ac5-b784-ea4bdeb61f67","Type":"ContainerDied","Data":"691b34bec4be7b788037a523724a68cd9d07f2a5e3cf4752fd744dcdcc82f313"} Oct 30 20:30:22.996338 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:22.996306 5755 generic.go:356] "Generic (PLEG): container finished" podID="80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e" containerID="b261c567719d62ad24f4d6a6b910b81b884ac48532891a97823e0b958cb6ee81" exitCode=0 Oct 30 20:30:22.996773 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:22.996408 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" event={"ID":"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e","Type":"ContainerDied","Data":"b261c567719d62ad24f4d6a6b910b81b884ac48532891a97823e0b958cb6ee81"} Oct 30 20:30:22.998987 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:22.998076 5755 generic.go:356] "Generic (PLEG): container finished" podID="a94074a3-21bf-48f3-9c01-a7ed53667bd0" containerID="d86744e50747d0cd977e0959d9e49d7f4d66e11aae1f89c3bfa425ca319b405b" exitCode=0 Oct 30 20:30:22.998987 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:22.998147 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" event={"ID":"a94074a3-21bf-48f3-9c01-a7ed53667bd0","Type":"ContainerDied","Data":"d86744e50747d0cd977e0959d9e49d7f4d66e11aae1f89c3bfa425ca319b405b"} Oct 30 20:30:24.004556 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:24.004524 5755 generic.go:356] "Generic (PLEG): container finished" podID="c094def6-82c7-4ac5-b784-ea4bdeb61f67" containerID="504648c0ee50a7af123f6110d47237e9d8413d6f6df92724d8da106248a8dc97" exitCode=0 Oct 30 20:30:24.005020 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:24.004638 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" event={"ID":"c094def6-82c7-4ac5-b784-ea4bdeb61f67","Type":"ContainerDied","Data":"504648c0ee50a7af123f6110d47237e9d8413d6f6df92724d8da106248a8dc97"} Oct 30 20:30:24.013350 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:24.013212 5755 generic.go:356] "Generic (PLEG): container finished" podID="4b1268c9-2572-4d90-8f9e-551e7747b302" containerID="48cd57b3cbf7133bac02cfc7445064015fe56f2df600f43a96dd011ef3290e31" exitCode=0 Oct 30 20:30:24.013350 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:24.013244 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" event={"ID":"4b1268c9-2572-4d90-8f9e-551e7747b302","Type":"ContainerDied","Data":"48cd57b3cbf7133bac02cfc7445064015fe56f2df600f43a96dd011ef3290e31"} Oct 30 20:30:25.021512 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:25.021426 5755 generic.go:356] "Generic (PLEG): container finished" podID="240d6e3e-81fe-4415-b581-a7439d0adab0" containerID="e9cea64781e60f4e1a7cd28dc8bb84684a2c0532ae356ac833abaf164311a463" exitCode=0 Oct 30 20:30:25.022186 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:25.022161 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" event={"ID":"240d6e3e-81fe-4415-b581-a7439d0adab0","Type":"ContainerDied","Data":"e9cea64781e60f4e1a7cd28dc8bb84684a2c0532ae356ac833abaf164311a463"} Oct 30 20:30:25.025480 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:25.025156 5755 generic.go:356] "Generic (PLEG): container finished" podID="a94074a3-21bf-48f3-9c01-a7ed53667bd0" containerID="934a50ea6de2aca2984988150810fb137bf73f97cc0c08dbe8c9c1f5c5ddac4e" exitCode=0 Oct 30 20:30:25.025480 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:25.025206 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" event={"ID":"a94074a3-21bf-48f3-9c01-a7ed53667bd0","Type":"ContainerDied","Data":"934a50ea6de2aca2984988150810fb137bf73f97cc0c08dbe8c9c1f5c5ddac4e"} Oct 30 20:30:32.081329 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:32.080302 5755 generic.go:356] "Generic (PLEG): container finished" podID="940b9fdc-1f1e-44f4-b718-517956fa735e" containerID="071b3508a6f4b2b396f90443253b15acfa8f4348208334ec49b70e9c4a2f9265" exitCode=0 Oct 30 20:30:32.081329 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:32.080431 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" event={"ID":"940b9fdc-1f1e-44f4-b718-517956fa735e","Type":"ContainerDied","Data":"071b3508a6f4b2b396f90443253b15acfa8f4348208334ec49b70e9c4a2f9265"} Oct 30 20:30:35.114778 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:35.114738 5755 generic.go:356] "Generic (PLEG): container finished" podID="80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e" containerID="dfe69cdc13b3b2ca03e517ab03bdfec7b2c7dcdf917e105ac4c58c8028c338a5" exitCode=0 Oct 30 20:30:35.115223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:35.114824 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" event={"ID":"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e","Type":"ContainerDied","Data":"dfe69cdc13b3b2ca03e517ab03bdfec7b2c7dcdf917e105ac4c58c8028c338a5"} Oct 30 20:30:36.312425 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:36.312359 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" Oct 30 20:30:36.419347 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:36.419120 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-bundle\") pod \"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e\" (UID: \"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e\") " Oct 30 20:30:36.419347 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:36.419174 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-util\" (UniqueName: \"kubernetes.io/empty-dir/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-util\") pod \"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e\" (UID: \"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e\") " Oct 30 20:30:36.419641 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:36.419612 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-kube-api-access-rt6wp\" (UniqueName: \"kubernetes.io/projected/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-kube-api-access-rt6wp\") pod \"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e\" (UID: \"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e\") " Oct 30 20:30:36.422511 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:36.422441 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-bundle" pod "80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e" (UID: "80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:36.422511 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:36.422445 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-kube-api-access-rt6wp" pod "80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e" (UID: "80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e"). InnerVolumeSpecName "kube-api-access-rt6wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:36.427336 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:36.427316 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-util" pod "80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e" (UID: "80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:36.520611 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:36.520578 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt6wp\" (UniqueName: \"kubernetes.io/projected/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-kube-api-access-rt6wp\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:36.520806 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:36.520615 5755 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-bundle\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:36.520806 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:36.520630 5755 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e-util\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:37.129212 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:37.129174 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" event={"ID":"c094def6-82c7-4ac5-b784-ea4bdeb61f67","Type":"ContainerStarted","Data":"8badc1bb36674637aa7b54cbad54c018868f5d01e4d172521f933b0900d279f3"} Oct 30 20:30:37.145511 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:37.145487 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" Oct 30 20:30:37.145657 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:37.145507 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n92qg" event={"ID":"80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e","Type":"ContainerDied","Data":"d8b38aa00e3041b5818a019420d45dfbd7c0ef0ababb02097e957dc0374ccf4d"} Oct 30 20:30:37.145657 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:37.145547 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8b38aa00e3041b5818a019420d45dfbd7c0ef0ababb02097e957dc0374ccf4d" Oct 30 20:30:37.150099 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:37.149934 5755 generic.go:356] "Generic (PLEG): container finished" podID="4b1268c9-2572-4d90-8f9e-551e7747b302" containerID="c63ee33abc1bddb222286a258aae429b2a0c5d3645d55d4b5e49de852db893d6" exitCode=0 Oct 30 20:30:37.150099 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:37.149986 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" event={"ID":"4b1268c9-2572-4d90-8f9e-551e7747b302","Type":"ContainerDied","Data":"c63ee33abc1bddb222286a258aae429b2a0c5d3645d55d4b5e49de852db893d6"} Oct 30 20:30:37.250091 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:37.250030 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" podStartSLOduration=2.493667302 podStartE2EDuration="29.250013586s" totalImagesPullingTime="26.756346284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:37.248889439 +0000 UTC m=+200.175446383" watchObservedRunningTime="2026-10-30 20:30:37.250013586 +0000 UTC m=+200.176570513" Oct 30 20:30:38.158550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.158453 5755 generic.go:356] "Generic (PLEG): container finished" podID="c094def6-82c7-4ac5-b784-ea4bdeb61f67" containerID="8badc1bb36674637aa7b54cbad54c018868f5d01e4d172521f933b0900d279f3" exitCode=0 Oct 30 20:30:38.158992 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.158589 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" event={"ID":"c094def6-82c7-4ac5-b784-ea4bdeb61f67","Type":"ContainerDied","Data":"8badc1bb36674637aa7b54cbad54c018868f5d01e4d172521f933b0900d279f3"} Oct 30 20:30:38.162307 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.162278 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" event={"ID":"a94074a3-21bf-48f3-9c01-a7ed53667bd0","Type":"ContainerStarted","Data":"2198582a1c45db81b85c8892be87389a4c45a0b01045e154b7c700f258fdd597"} Oct 30 20:30:38.164703 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.164678 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" event={"ID":"240d6e3e-81fe-4415-b581-a7439d0adab0","Type":"ContainerStarted","Data":"4faaed5e1bee959075e51b493a21cca6dff282b51a9bb5730671d816f48040b9"} Oct 30 20:30:38.315056 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.315034 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" Oct 30 20:30:38.395207 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.395157 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" podStartSLOduration=3.072125168 podStartE2EDuration="30.395139195s" totalImagesPullingTime="27.323014027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:38.393828134 +0000 UTC m=+201.320385067" watchObservedRunningTime="2026-10-30 20:30:38.395139195 +0000 UTC m=+201.321696111" Oct 30 20:30:38.419923 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.419870 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-nfvcr"] Oct 30 20:30:38.420160 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.420149 5755 cpu_manager.go:414] "containerMap: removing container" podUID="80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e" containerName="extract" Oct 30 20:30:38.420211 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.420161 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e" containerName="extract" Oct 30 20:30:38.420211 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.420171 5755 cpu_manager.go:414] "containerMap: removing container" podUID="80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e" containerName="util" Oct 30 20:30:38.420211 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.420175 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e" containerName="util" Oct 30 20:30:38.420211 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.420185 5755 cpu_manager.go:414] "containerMap: removing container" podUID="80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e" containerName="pull" Oct 30 20:30:38.420211 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.420189 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e" containerName="pull" Oct 30 20:30:38.420211 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.420196 5755 cpu_manager.go:414] "containerMap: removing container" podUID="4b1268c9-2572-4d90-8f9e-551e7747b302" containerName="extract" Oct 30 20:30:38.420211 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.420200 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1268c9-2572-4d90-8f9e-551e7747b302" containerName="extract" Oct 30 20:30:38.420211 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.420208 5755 cpu_manager.go:414] "containerMap: removing container" podUID="4b1268c9-2572-4d90-8f9e-551e7747b302" containerName="util" Oct 30 20:30:38.420211 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.420212 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1268c9-2572-4d90-8f9e-551e7747b302" containerName="util" Oct 30 20:30:38.420508 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.420226 5755 cpu_manager.go:414] "containerMap: removing container" podUID="4b1268c9-2572-4d90-8f9e-551e7747b302" containerName="pull" Oct 30 20:30:38.420508 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.420230 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1268c9-2572-4d90-8f9e-551e7747b302" containerName="pull" Oct 30 20:30:38.420508 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.420277 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="80dc99f7-a894-4dd2-86c3-4a6e7d4ab99e" containerName="extract" Oct 30 20:30:38.420508 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.420284 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1268c9-2572-4d90-8f9e-551e7747b302" containerName="extract" Oct 30 20:30:38.421902 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.421888 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nfvcr" Oct 30 20:30:38.436288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.436264 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1268c9-2572-4d90-8f9e-551e7747b302-util\" (UniqueName: \"kubernetes.io/empty-dir/4b1268c9-2572-4d90-8f9e-551e7747b302-util\") pod \"4b1268c9-2572-4d90-8f9e-551e7747b302\" (UID: \"4b1268c9-2572-4d90-8f9e-551e7747b302\") " Oct 30 20:30:38.436411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.436372 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1268c9-2572-4d90-8f9e-551e7747b302-kube-api-access-rg7nv\" (UniqueName: \"kubernetes.io/projected/4b1268c9-2572-4d90-8f9e-551e7747b302-kube-api-access-rg7nv\") pod \"4b1268c9-2572-4d90-8f9e-551e7747b302\" (UID: \"4b1268c9-2572-4d90-8f9e-551e7747b302\") " Oct 30 20:30:38.436479 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.436445 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1268c9-2572-4d90-8f9e-551e7747b302-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b1268c9-2572-4d90-8f9e-551e7747b302-bundle\") pod \"4b1268c9-2572-4d90-8f9e-551e7747b302\" (UID: \"4b1268c9-2572-4d90-8f9e-551e7747b302\") " Oct 30 20:30:38.436881 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.436848 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1268c9-2572-4d90-8f9e-551e7747b302-bundle" pod "4b1268c9-2572-4d90-8f9e-551e7747b302" (UID: "4b1268c9-2572-4d90-8f9e-551e7747b302"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:38.438375 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.438358 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1268c9-2572-4d90-8f9e-551e7747b302-kube-api-access-rg7nv" pod "4b1268c9-2572-4d90-8f9e-551e7747b302" (UID: "4b1268c9-2572-4d90-8f9e-551e7747b302"). InnerVolumeSpecName "kube-api-access-rg7nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:38.441352 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.441330 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1268c9-2572-4d90-8f9e-551e7747b302-util" pod "4b1268c9-2572-4d90-8f9e-551e7747b302" (UID: "4b1268c9-2572-4d90-8f9e-551e7747b302"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:38.475468 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.475444 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 20:30:38.475468 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.475454 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:30:38.475606 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.475511 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-5xqqz\"" Oct 30 20:30:38.506512 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.506488 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-nfvcr"] Oct 30 20:30:38.510523 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.510441 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" podStartSLOduration=3.052810343 podStartE2EDuration="30.510425985s" totalImagesPullingTime="27.457615642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:38.509930346 +0000 UTC m=+201.436487280" watchObservedRunningTime="2026-10-30 20:30:38.510425985 +0000 UTC m=+201.436982901" Oct 30 20:30:38.537276 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.537244 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5nbk\" (UniqueName: \"kubernetes.io/projected/aaa2b81c-fa84-4fda-84d6-d06504dc6c0b-kube-api-access-s5nbk\") pod \"jobset-operator-674cd59dc5-nfvcr\" (UID: \"aaa2b81c-fa84-4fda-84d6-d06504dc6c0b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nfvcr" Oct 30 20:30:38.537424 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.537378 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaa2b81c-fa84-4fda-84d6-d06504dc6c0b-tmp\") pod \"jobset-operator-674cd59dc5-nfvcr\" (UID: \"aaa2b81c-fa84-4fda-84d6-d06504dc6c0b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nfvcr" Oct 30 20:30:38.537473 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.537461 5755 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b1268c9-2572-4d90-8f9e-551e7747b302-util\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:38.537516 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.537474 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg7nv\" (UniqueName: \"kubernetes.io/projected/4b1268c9-2572-4d90-8f9e-551e7747b302-kube-api-access-rg7nv\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:38.537516 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.537483 5755 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b1268c9-2572-4d90-8f9e-551e7747b302-bundle\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:38.638476 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.638445 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaa2b81c-fa84-4fda-84d6-d06504dc6c0b-tmp\") pod \"jobset-operator-674cd59dc5-nfvcr\" (UID: \"aaa2b81c-fa84-4fda-84d6-d06504dc6c0b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nfvcr" Oct 30 20:30:38.638647 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.638492 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5nbk\" (UniqueName: \"kubernetes.io/projected/aaa2b81c-fa84-4fda-84d6-d06504dc6c0b-kube-api-access-s5nbk\") pod \"jobset-operator-674cd59dc5-nfvcr\" (UID: \"aaa2b81c-fa84-4fda-84d6-d06504dc6c0b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nfvcr" Oct 30 20:30:38.638853 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.638832 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaa2b81c-fa84-4fda-84d6-d06504dc6c0b-tmp\") pod \"jobset-operator-674cd59dc5-nfvcr\" (UID: \"aaa2b81c-fa84-4fda-84d6-d06504dc6c0b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nfvcr" Oct 30 20:30:38.697886 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.697829 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5nbk\" (UniqueName: \"kubernetes.io/projected/aaa2b81c-fa84-4fda-84d6-d06504dc6c0b-kube-api-access-s5nbk\") pod \"jobset-operator-674cd59dc5-nfvcr\" (UID: \"aaa2b81c-fa84-4fda-84d6-d06504dc6c0b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nfvcr" Oct 30 20:30:38.769371 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:38.769338 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nfvcr" Oct 30 20:30:39.137039 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.137015 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-nfvcr"] Oct 30 20:30:39.170641 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.170620 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" Oct 30 20:30:39.171002 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.170623 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vz2mb" event={"ID":"4b1268c9-2572-4d90-8f9e-551e7747b302","Type":"ContainerDied","Data":"8537b20df13004ac90c7fb1ef05ecdd1d730172ff5ecda6dc454fef56c73488f"} Oct 30 20:30:39.171002 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.170747 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8537b20df13004ac90c7fb1ef05ecdd1d730172ff5ecda6dc454fef56c73488f" Oct 30 20:30:39.172954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.172927 5755 generic.go:356] "Generic (PLEG): container finished" podID="a94074a3-21bf-48f3-9c01-a7ed53667bd0" containerID="2198582a1c45db81b85c8892be87389a4c45a0b01045e154b7c700f258fdd597" exitCode=0 Oct 30 20:30:39.173072 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.173014 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" event={"ID":"a94074a3-21bf-48f3-9c01-a7ed53667bd0","Type":"ContainerDied","Data":"2198582a1c45db81b85c8892be87389a4c45a0b01045e154b7c700f258fdd597"} Oct 30 20:30:39.175260 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.175233 5755 generic.go:356] "Generic (PLEG): container finished" podID="240d6e3e-81fe-4415-b581-a7439d0adab0" containerID="4faaed5e1bee959075e51b493a21cca6dff282b51a9bb5730671d816f48040b9" exitCode=0 Oct 30 20:30:39.175357 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.175313 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" event={"ID":"240d6e3e-81fe-4415-b581-a7439d0adab0","Type":"ContainerDied","Data":"4faaed5e1bee959075e51b493a21cca6dff282b51a9bb5730671d816f48040b9"} Oct 30 20:30:39.177184 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.177166 5755 generic.go:356] "Generic (PLEG): container finished" podID="940b9fdc-1f1e-44f4-b718-517956fa735e" containerID="299b24c7f71e4e41f4136b931d9a7650798dd7742e9399c643b6e141718a6e8b" exitCode=0 Oct 30 20:30:39.177257 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.177235 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" event={"ID":"940b9fdc-1f1e-44f4-b718-517956fa735e","Type":"ContainerDied","Data":"299b24c7f71e4e41f4136b931d9a7650798dd7742e9399c643b6e141718a6e8b"} Oct 30 20:30:39.206776 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:30:39.206706 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaaa2b81c_fa84_4fda_84d6_d06504dc6c0b.slice/crio-a9713ae0de8610b9fc5fa967dec71d2c324ed320894bf47bae30635a1d4a5a7b WatchSource:0}: Error finding container a9713ae0de8610b9fc5fa967dec71d2c324ed320894bf47bae30635a1d4a5a7b: Status 404 returned error can't find the container with id a9713ae0de8610b9fc5fa967dec71d2c324ed320894bf47bae30635a1d4a5a7b Oct 30 20:30:39.288565 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.288545 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" Oct 30 20:30:39.445596 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.445562 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c094def6-82c7-4ac5-b784-ea4bdeb61f67-util\" (UniqueName: \"kubernetes.io/empty-dir/c094def6-82c7-4ac5-b784-ea4bdeb61f67-util\") pod \"c094def6-82c7-4ac5-b784-ea4bdeb61f67\" (UID: \"c094def6-82c7-4ac5-b784-ea4bdeb61f67\") " Oct 30 20:30:39.445781 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.445619 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c094def6-82c7-4ac5-b784-ea4bdeb61f67-kube-api-access-wkq7f\" (UniqueName: \"kubernetes.io/projected/c094def6-82c7-4ac5-b784-ea4bdeb61f67-kube-api-access-wkq7f\") pod \"c094def6-82c7-4ac5-b784-ea4bdeb61f67\" (UID: \"c094def6-82c7-4ac5-b784-ea4bdeb61f67\") " Oct 30 20:30:39.445781 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.445646 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c094def6-82c7-4ac5-b784-ea4bdeb61f67-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c094def6-82c7-4ac5-b784-ea4bdeb61f67-bundle\") pod \"c094def6-82c7-4ac5-b784-ea4bdeb61f67\" (UID: \"c094def6-82c7-4ac5-b784-ea4bdeb61f67\") " Oct 30 20:30:39.446277 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.446249 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c094def6-82c7-4ac5-b784-ea4bdeb61f67-bundle" pod "c094def6-82c7-4ac5-b784-ea4bdeb61f67" (UID: "c094def6-82c7-4ac5-b784-ea4bdeb61f67"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:39.447763 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.447745 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c094def6-82c7-4ac5-b784-ea4bdeb61f67-kube-api-access-wkq7f" pod "c094def6-82c7-4ac5-b784-ea4bdeb61f67" (UID: "c094def6-82c7-4ac5-b784-ea4bdeb61f67"). InnerVolumeSpecName "kube-api-access-wkq7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:39.451716 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.451695 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c094def6-82c7-4ac5-b784-ea4bdeb61f67-util" pod "c094def6-82c7-4ac5-b784-ea4bdeb61f67" (UID: "c094def6-82c7-4ac5-b784-ea4bdeb61f67"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:39.546529 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.546498 5755 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c094def6-82c7-4ac5-b784-ea4bdeb61f67-util\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:39.546529 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.546523 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkq7f\" (UniqueName: \"kubernetes.io/projected/c094def6-82c7-4ac5-b784-ea4bdeb61f67-kube-api-access-wkq7f\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:39.546529 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:39.546532 5755 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c094def6-82c7-4ac5-b784-ea4bdeb61f67-bundle\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:40.182989 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.182961 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" Oct 30 20:30:40.182989 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.182974 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljb8w" event={"ID":"c094def6-82c7-4ac5-b784-ea4bdeb61f67","Type":"ContainerDied","Data":"8a419a569b8b563efde82f79de741b15809807795ce4ad293ed6441be671571e"} Oct 30 20:30:40.183420 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.183031 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a419a569b8b563efde82f79de741b15809807795ce4ad293ed6441be671571e" Oct 30 20:30:40.184340 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.184317 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nfvcr" event={"ID":"aaa2b81c-fa84-4fda-84d6-d06504dc6c0b","Type":"ContainerStarted","Data":"a9713ae0de8610b9fc5fa967dec71d2c324ed320894bf47bae30635a1d4a5a7b"} Oct 30 20:30:40.321477 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.321455 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" Oct 30 20:30:40.335871 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.335850 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" Oct 30 20:30:40.359870 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.359849 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" Oct 30 20:30:40.454979 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.454915 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240d6e3e-81fe-4415-b581-a7439d0adab0-util\" (UniqueName: \"kubernetes.io/empty-dir/240d6e3e-81fe-4415-b581-a7439d0adab0-util\") pod \"240d6e3e-81fe-4415-b581-a7439d0adab0\" (UID: \"240d6e3e-81fe-4415-b581-a7439d0adab0\") " Oct 30 20:30:40.454979 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.454963 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240d6e3e-81fe-4415-b581-a7439d0adab0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/240d6e3e-81fe-4415-b581-a7439d0adab0-bundle\") pod \"240d6e3e-81fe-4415-b581-a7439d0adab0\" (UID: \"240d6e3e-81fe-4415-b581-a7439d0adab0\") " Oct 30 20:30:40.455167 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.455013 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940b9fdc-1f1e-44f4-b718-517956fa735e-kube-api-access-gmf89\" (UniqueName: \"kubernetes.io/projected/940b9fdc-1f1e-44f4-b718-517956fa735e-kube-api-access-gmf89\") pod \"940b9fdc-1f1e-44f4-b718-517956fa735e\" (UID: \"940b9fdc-1f1e-44f4-b718-517956fa735e\") " Oct 30 20:30:40.455167 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.455042 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940b9fdc-1f1e-44f4-b718-517956fa735e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/940b9fdc-1f1e-44f4-b718-517956fa735e-bundle\") pod \"940b9fdc-1f1e-44f4-b718-517956fa735e\" (UID: \"940b9fdc-1f1e-44f4-b718-517956fa735e\") " Oct 30 20:30:40.455167 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.455077 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/240d6e3e-81fe-4415-b581-a7439d0adab0-kube-api-access-2dbcm\" (UniqueName: \"kubernetes.io/projected/240d6e3e-81fe-4415-b581-a7439d0adab0-kube-api-access-2dbcm\") pod \"240d6e3e-81fe-4415-b581-a7439d0adab0\" (UID: \"240d6e3e-81fe-4415-b581-a7439d0adab0\") " Oct 30 20:30:40.455167 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.455111 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a94074a3-21bf-48f3-9c01-a7ed53667bd0-util\" (UniqueName: \"kubernetes.io/empty-dir/a94074a3-21bf-48f3-9c01-a7ed53667bd0-util\") pod \"a94074a3-21bf-48f3-9c01-a7ed53667bd0\" (UID: \"a94074a3-21bf-48f3-9c01-a7ed53667bd0\") " Oct 30 20:30:40.455167 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.455150 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a94074a3-21bf-48f3-9c01-a7ed53667bd0-kube-api-access-r7bq9\" (UniqueName: \"kubernetes.io/projected/a94074a3-21bf-48f3-9c01-a7ed53667bd0-kube-api-access-r7bq9\") pod \"a94074a3-21bf-48f3-9c01-a7ed53667bd0\" (UID: \"a94074a3-21bf-48f3-9c01-a7ed53667bd0\") " Oct 30 20:30:40.455434 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.455203 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940b9fdc-1f1e-44f4-b718-517956fa735e-util\" (UniqueName: \"kubernetes.io/empty-dir/940b9fdc-1f1e-44f4-b718-517956fa735e-util\") pod \"940b9fdc-1f1e-44f4-b718-517956fa735e\" (UID: \"940b9fdc-1f1e-44f4-b718-517956fa735e\") " Oct 30 20:30:40.455434 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.455274 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a94074a3-21bf-48f3-9c01-a7ed53667bd0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a94074a3-21bf-48f3-9c01-a7ed53667bd0-bundle\") pod \"a94074a3-21bf-48f3-9c01-a7ed53667bd0\" (UID: \"a94074a3-21bf-48f3-9c01-a7ed53667bd0\") " Oct 30 20:30:40.455612 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.455584 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240d6e3e-81fe-4415-b581-a7439d0adab0-bundle" pod "240d6e3e-81fe-4415-b581-a7439d0adab0" (UID: "240d6e3e-81fe-4415-b581-a7439d0adab0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:40.456295 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.456222 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940b9fdc-1f1e-44f4-b718-517956fa735e-bundle" pod "940b9fdc-1f1e-44f4-b718-517956fa735e" (UID: "940b9fdc-1f1e-44f4-b718-517956fa735e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:40.456467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.456367 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94074a3-21bf-48f3-9c01-a7ed53667bd0-bundle" pod "a94074a3-21bf-48f3-9c01-a7ed53667bd0" (UID: "a94074a3-21bf-48f3-9c01-a7ed53667bd0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:40.458288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.458255 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94074a3-21bf-48f3-9c01-a7ed53667bd0-kube-api-access-r7bq9" pod "a94074a3-21bf-48f3-9c01-a7ed53667bd0" (UID: "a94074a3-21bf-48f3-9c01-a7ed53667bd0"). InnerVolumeSpecName "kube-api-access-r7bq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:40.458418 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.458398 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940b9fdc-1f1e-44f4-b718-517956fa735e-kube-api-access-gmf89" pod "940b9fdc-1f1e-44f4-b718-517956fa735e" (UID: "940b9fdc-1f1e-44f4-b718-517956fa735e"). InnerVolumeSpecName "kube-api-access-gmf89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:40.459486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.459464 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240d6e3e-81fe-4415-b581-a7439d0adab0-kube-api-access-2dbcm" pod "240d6e3e-81fe-4415-b581-a7439d0adab0" (UID: "240d6e3e-81fe-4415-b581-a7439d0adab0"). InnerVolumeSpecName "kube-api-access-2dbcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:40.459885 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.459868 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240d6e3e-81fe-4415-b581-a7439d0adab0-util" pod "240d6e3e-81fe-4415-b581-a7439d0adab0" (UID: "240d6e3e-81fe-4415-b581-a7439d0adab0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:40.464248 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.464218 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940b9fdc-1f1e-44f4-b718-517956fa735e-util" pod "940b9fdc-1f1e-44f4-b718-517956fa735e" (UID: "940b9fdc-1f1e-44f4-b718-517956fa735e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:40.464574 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.464557 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94074a3-21bf-48f3-9c01-a7ed53667bd0-util" pod "a94074a3-21bf-48f3-9c01-a7ed53667bd0" (UID: "a94074a3-21bf-48f3-9c01-a7ed53667bd0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:40.555840 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.555811 5755 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/240d6e3e-81fe-4415-b581-a7439d0adab0-util\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:40.555840 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.555833 5755 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/240d6e3e-81fe-4415-b581-a7439d0adab0-bundle\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:40.555840 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.555843 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmf89\" (UniqueName: \"kubernetes.io/projected/940b9fdc-1f1e-44f4-b718-517956fa735e-kube-api-access-gmf89\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:40.556040 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.555852 5755 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/940b9fdc-1f1e-44f4-b718-517956fa735e-bundle\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:40.556040 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.555862 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dbcm\" (UniqueName: \"kubernetes.io/projected/240d6e3e-81fe-4415-b581-a7439d0adab0-kube-api-access-2dbcm\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:40.556040 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.555870 5755 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a94074a3-21bf-48f3-9c01-a7ed53667bd0-util\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:40.556040 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.555879 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7bq9\" (UniqueName: \"kubernetes.io/projected/a94074a3-21bf-48f3-9c01-a7ed53667bd0-kube-api-access-r7bq9\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:40.556040 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.555887 5755 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/940b9fdc-1f1e-44f4-b718-517956fa735e-util\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:40.556040 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:40.555895 5755 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a94074a3-21bf-48f3-9c01-a7ed53667bd0-bundle\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:30:41.192409 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:41.192363 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" Oct 30 20:30:41.192916 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:41.192364 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2k5dk" event={"ID":"940b9fdc-1f1e-44f4-b718-517956fa735e","Type":"ContainerDied","Data":"49a24dfd26250f58442ed7791a7b751aab5e3a223174a6fe88a2d8dae0235b68"} Oct 30 20:30:41.192916 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:41.192584 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49a24dfd26250f58442ed7791a7b751aab5e3a223174a6fe88a2d8dae0235b68" Oct 30 20:30:41.202784 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:41.202753 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" event={"ID":"a94074a3-21bf-48f3-9c01-a7ed53667bd0","Type":"ContainerDied","Data":"b5fc75e8c29834da1ba6b1b63ec992b1d1fd8b7adeea8171928e756c5281f5d8"} Oct 30 20:30:41.202889 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:41.202794 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbw958" Oct 30 20:30:41.202889 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:41.202801 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5fc75e8c29834da1ba6b1b63ec992b1d1fd8b7adeea8171928e756c5281f5d8" Oct 30 20:30:41.211321 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:41.208282 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" Oct 30 20:30:41.213203 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:41.213158 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s9c84" event={"ID":"240d6e3e-81fe-4415-b581-a7439d0adab0","Type":"ContainerDied","Data":"17228da55502689c0fbc828bef44b2e3b0e3564a691b7f1bf8219b67ba3d8157"} Oct 30 20:30:41.213280 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:41.213225 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17228da55502689c0fbc828bef44b2e3b0e3564a691b7f1bf8219b67ba3d8157" Oct 30 20:30:41.263226 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:30:41.263193 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:30:43.620182 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620151 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc"] Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620433 5755 cpu_manager.go:414] "containerMap: removing container" podUID="240d6e3e-81fe-4415-b581-a7439d0adab0" containerName="util" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620446 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240d6e3e-81fe-4415-b581-a7439d0adab0" containerName="util" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620463 5755 cpu_manager.go:414] "containerMap: removing container" podUID="c094def6-82c7-4ac5-b784-ea4bdeb61f67" containerName="pull" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620468 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c094def6-82c7-4ac5-b784-ea4bdeb61f67" containerName="pull" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620474 5755 cpu_manager.go:414] "containerMap: removing container" podUID="940b9fdc-1f1e-44f4-b718-517956fa735e" containerName="pull" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620480 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940b9fdc-1f1e-44f4-b718-517956fa735e" containerName="pull" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620490 5755 cpu_manager.go:414] "containerMap: removing container" podUID="a94074a3-21bf-48f3-9c01-a7ed53667bd0" containerName="util" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620495 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94074a3-21bf-48f3-9c01-a7ed53667bd0" containerName="util" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620500 5755 cpu_manager.go:414] "containerMap: removing container" podUID="c094def6-82c7-4ac5-b784-ea4bdeb61f67" containerName="extract" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620504 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c094def6-82c7-4ac5-b784-ea4bdeb61f67" containerName="extract" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620510 5755 cpu_manager.go:414] "containerMap: removing container" podUID="240d6e3e-81fe-4415-b581-a7439d0adab0" containerName="pull" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620514 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240d6e3e-81fe-4415-b581-a7439d0adab0" containerName="pull" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620520 5755 cpu_manager.go:414] "containerMap: removing container" podUID="240d6e3e-81fe-4415-b581-a7439d0adab0" containerName="extract" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620524 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240d6e3e-81fe-4415-b581-a7439d0adab0" containerName="extract" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620536 5755 cpu_manager.go:414] "containerMap: removing container" podUID="940b9fdc-1f1e-44f4-b718-517956fa735e" containerName="util" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620541 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940b9fdc-1f1e-44f4-b718-517956fa735e" containerName="util" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620547 5755 cpu_manager.go:414] "containerMap: removing container" podUID="c094def6-82c7-4ac5-b784-ea4bdeb61f67" containerName="util" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620551 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c094def6-82c7-4ac5-b784-ea4bdeb61f67" containerName="util" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620561 5755 cpu_manager.go:414] "containerMap: removing container" podUID="a94074a3-21bf-48f3-9c01-a7ed53667bd0" containerName="pull" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620565 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94074a3-21bf-48f3-9c01-a7ed53667bd0" containerName="pull" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620570 5755 cpu_manager.go:414] "containerMap: removing container" podUID="a94074a3-21bf-48f3-9c01-a7ed53667bd0" containerName="extract" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620574 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94074a3-21bf-48f3-9c01-a7ed53667bd0" containerName="extract" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620580 5755 cpu_manager.go:414] "containerMap: removing container" podUID="940b9fdc-1f1e-44f4-b718-517956fa735e" containerName="extract" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620584 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940b9fdc-1f1e-44f4-b718-517956fa735e" containerName="extract" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620624 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="c094def6-82c7-4ac5-b784-ea4bdeb61f67" containerName="extract" Oct 30 20:30:43.620618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620630 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="240d6e3e-81fe-4415-b581-a7439d0adab0" containerName="extract" Oct 30 20:30:43.621452 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620638 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="940b9fdc-1f1e-44f4-b718-517956fa735e" containerName="extract" Oct 30 20:30:43.621452 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.620642 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="a94074a3-21bf-48f3-9c01-a7ed53667bd0" containerName="extract" Oct 30 20:30:43.641467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.641443 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.758268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.758239 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 20:30:43.758268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.758240 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 20:30:43.758514 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.758381 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 20:30:43.758567 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.758531 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 20:30:43.758635 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.758621 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jjpq2\"" Oct 30 20:30:43.758684 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.758670 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 20:30:43.761371 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.761347 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc"] Oct 30 20:30:43.781793 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.781769 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-manifests\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.781905 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.781797 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-metrics-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.781958 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.781899 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx9gx\" (UniqueName: \"kubernetes.io/projected/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-kube-api-access-bx9gx\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.782004 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.781990 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-webhook-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.782050 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.782017 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-tmp\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.782050 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.782040 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.883249 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.883215 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx9gx\" (UniqueName: \"kubernetes.io/projected/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-kube-api-access-bx9gx\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.883249 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.883252 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-webhook-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.883474 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.883270 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-tmp\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.883474 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.883421 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.883474 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.883467 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-manifests\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.883628 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.883546 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-metrics-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.883794 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.883775 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-manifests\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.885830 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.885804 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-tmp\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.886093 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.885979 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.886093 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.886021 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-webhook-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.886216 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.886124 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-metrics-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.941610 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.941584 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx9gx\" (UniqueName: \"kubernetes.io/projected/7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c-kube-api-access-bx9gx\") pod \"opendatahub-operator-controller-manager-fdf79688b-nd9wc\" (UID: \"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:43.988603 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:43.988582 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:44.223927 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:44.223892 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nfvcr" event={"ID":"aaa2b81c-fa84-4fda-84d6-d06504dc6c0b","Type":"ContainerStarted","Data":"310236c603cccae99d6de75f13fa3dcbbf4ed2ad2f51ae881fb664a0e2682e4a"} Oct 30 20:30:44.338518 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:44.338375 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nfvcr" podStartSLOduration=2.09185312 podStartE2EDuration="6.338360199s" totalImagesPullingTime="4.246507079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:44.337632536 +0000 UTC m=+207.264189471" watchObservedRunningTime="2026-10-30 20:30:44.338360199 +0000 UTC m=+207.264917107" Oct 30 20:30:44.541742 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:44.541717 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc"] Oct 30 20:30:44.544006 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:30:44.543978 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb76ad8_11f8_459f_b9d0_cfa0e0717e0c.slice/crio-6068b86f8ba93f1dd96e966859aa4cd70e28fad2e61cd8c563a11922a4d7d621 WatchSource:0}: Error finding container 6068b86f8ba93f1dd96e966859aa4cd70e28fad2e61cd8c563a11922a4d7d621: Status 404 returned error can't find the container with id 6068b86f8ba93f1dd96e966859aa4cd70e28fad2e61cd8c563a11922a4d7d621 Oct 30 20:30:45.228717 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:45.228676 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" event={"ID":"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c","Type":"ContainerStarted","Data":"6068b86f8ba93f1dd96e966859aa4cd70e28fad2e61cd8c563a11922a4d7d621"} Oct 30 20:30:48.246009 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:48.245971 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" event={"ID":"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c","Type":"ContainerStarted","Data":"8104c903f58dbb82658799f780c89d10f2c212c1d82409166b4a21d18452609a"} Oct 30 20:30:49.255885 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:49.255849 5755 generic.go:356] "Generic (PLEG): container finished" podID="7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c" containerID="8104c903f58dbb82658799f780c89d10f2c212c1d82409166b4a21d18452609a" exitCode=0 Oct 30 20:30:49.256355 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:49.255936 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" event={"ID":"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c","Type":"ContainerDied","Data":"8104c903f58dbb82658799f780c89d10f2c212c1d82409166b4a21d18452609a"} Oct 30 20:30:50.261484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:50.261445 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" event={"ID":"7fb76ad8-11f8-459f-b9d0-cfa0e0717e0c","Type":"ContainerStarted","Data":"f9c6c2b261875eb58e6928bdc0fae3fe53f2f954ecaaaf06b8dc93103fbe446b"} Oct 30 20:30:50.261924 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:50.261549 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:30:50.497301 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:30:50.497236 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" podStartSLOduration=2.8827089089999998 podStartE2EDuration="7.497219533s" totalImagesPullingTime="4.154267754s" totalInitContainerRuntime="460.24287ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:50.447027591 +0000 UTC m=+213.373584541" watchObservedRunningTime="2026-10-30 20:30:50.497219533 +0000 UTC m=+213.423776442" Oct 30 20:31:01.267057 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:01.267030 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-nd9wc" Oct 30 20:31:02.231006 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:02.230970 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-pjpr9"] Oct 30 20:31:02.232986 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:02.232971 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pjpr9" Oct 30 20:31:02.235980 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:02.235960 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzcpg\" (UniqueName: \"kubernetes.io/projected/c90843ee-3a81-471e-ba63-4c3ef73eea83-kube-api-access-zzcpg\") pod \"limitador-operator-controller-manager-5778bb4449-pjpr9\" (UID: \"c90843ee-3a81-471e-ba63-4c3ef73eea83\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pjpr9" Oct 30 20:31:02.290058 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:02.290035 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4nq6t\"" Oct 30 20:31:02.293411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:02.293362 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 20:31:02.293535 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:02.293455 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 20:31:02.294808 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:02.294788 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-pjpr9"] Oct 30 20:31:02.337397 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:02.337365 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzcpg\" (UniqueName: \"kubernetes.io/projected/c90843ee-3a81-471e-ba63-4c3ef73eea83-kube-api-access-zzcpg\") pod \"limitador-operator-controller-manager-5778bb4449-pjpr9\" (UID: \"c90843ee-3a81-471e-ba63-4c3ef73eea83\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pjpr9" Oct 30 20:31:02.398574 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:02.398549 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzcpg\" (UniqueName: \"kubernetes.io/projected/c90843ee-3a81-471e-ba63-4c3ef73eea83-kube-api-access-zzcpg\") pod \"limitador-operator-controller-manager-5778bb4449-pjpr9\" (UID: \"c90843ee-3a81-471e-ba63-4c3ef73eea83\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pjpr9" Oct 30 20:31:02.560276 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:02.560190 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pjpr9" Oct 30 20:31:02.981707 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:02.981681 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-pjpr9"] Oct 30 20:31:02.984189 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:31:02.984168 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90843ee_3a81_471e_ba63_4c3ef73eea83.slice/crio-f73291ae4d26013bd3c24f0801416d59671cc3e4ab12225ca9cb599a4b3f065c WatchSource:0}: Error finding container f73291ae4d26013bd3c24f0801416d59671cc3e4ab12225ca9cb599a4b3f065c: Status 404 returned error can't find the container with id f73291ae4d26013bd3c24f0801416d59671cc3e4ab12225ca9cb599a4b3f065c Oct 30 20:31:03.314445 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:03.314346 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pjpr9" event={"ID":"c90843ee-3a81-471e-ba63-4c3ef73eea83","Type":"ContainerStarted","Data":"f73291ae4d26013bd3c24f0801416d59671cc3e4ab12225ca9cb599a4b3f065c"} Oct 30 20:31:09.511657 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:09.511624 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-f8bxl"] Oct 30 20:31:09.513640 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:09.513625 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-f8bxl" Oct 30 20:31:09.614531 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:09.614498 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59vfl\" (UniqueName: \"kubernetes.io/projected/f2df5d57-9618-459b-b0fc-5a8fcfdde92e-kube-api-access-59vfl\") pod \"cert-manager-webhook-5dbbbf75cc-f8bxl\" (UID: \"f2df5d57-9618-459b-b0fc-5a8fcfdde92e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-f8bxl" Oct 30 20:31:09.614683 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:09.614567 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2df5d57-9618-459b-b0fc-5a8fcfdde92e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-f8bxl\" (UID: \"f2df5d57-9618-459b-b0fc-5a8fcfdde92e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-f8bxl" Oct 30 20:31:09.617139 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:09.617118 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 20:31:09.617193 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:09.617175 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 20:31:09.617362 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:09.617351 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7mzr5\"" Oct 30 20:31:09.619081 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:09.619065 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-f8bxl"] Oct 30 20:31:09.715403 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:09.715355 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2df5d57-9618-459b-b0fc-5a8fcfdde92e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-f8bxl\" (UID: \"f2df5d57-9618-459b-b0fc-5a8fcfdde92e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-f8bxl" Oct 30 20:31:09.715559 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:09.715407 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59vfl\" (UniqueName: \"kubernetes.io/projected/f2df5d57-9618-459b-b0fc-5a8fcfdde92e-kube-api-access-59vfl\") pod \"cert-manager-webhook-5dbbbf75cc-f8bxl\" (UID: \"f2df5d57-9618-459b-b0fc-5a8fcfdde92e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-f8bxl" Oct 30 20:31:09.775101 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:09.775042 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2df5d57-9618-459b-b0fc-5a8fcfdde92e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-f8bxl\" (UID: \"f2df5d57-9618-459b-b0fc-5a8fcfdde92e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-f8bxl" Oct 30 20:31:09.775200 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:09.775104 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59vfl\" (UniqueName: \"kubernetes.io/projected/f2df5d57-9618-459b-b0fc-5a8fcfdde92e-kube-api-access-59vfl\") pod \"cert-manager-webhook-5dbbbf75cc-f8bxl\" (UID: \"f2df5d57-9618-459b-b0fc-5a8fcfdde92e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-f8bxl" Oct 30 20:31:09.823872 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:09.823845 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-f8bxl" Oct 30 20:31:10.197117 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:10.197049 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-f8bxl"] Oct 30 20:31:10.200005 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:31:10.199978 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2df5d57_9618_459b_b0fc_5a8fcfdde92e.slice/crio-ebeb0efee4bdfc3e79eae3bae686bdd23cba7f76e4ec794ced3978225cd661d5 WatchSource:0}: Error finding container ebeb0efee4bdfc3e79eae3bae686bdd23cba7f76e4ec794ced3978225cd661d5: Status 404 returned error can't find the container with id ebeb0efee4bdfc3e79eae3bae686bdd23cba7f76e4ec794ced3978225cd661d5 Oct 30 20:31:10.344822 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:10.344786 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-f8bxl" event={"ID":"f2df5d57-9618-459b-b0fc-5a8fcfdde92e","Type":"ContainerStarted","Data":"ebeb0efee4bdfc3e79eae3bae686bdd23cba7f76e4ec794ced3978225cd661d5"} Oct 30 20:31:11.255436 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:31:11.255381 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:31:11.729888 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:11.729848 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj"] Oct 30 20:31:11.732937 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:11.732915 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj" Oct 30 20:31:11.789711 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:11.789683 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 20:31:11.790366 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:11.790343 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj"] Oct 30 20:31:11.833443 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:11.833409 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/18102e92-59b1-409c-b1cf-fd6b9c1a90b0-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jwmvj\" (UID: \"18102e92-59b1-409c-b1cf-fd6b9c1a90b0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj" Oct 30 20:31:11.833573 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:11.833447 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/18102e92-59b1-409c-b1cf-fd6b9c1a90b0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jwmvj\" (UID: \"18102e92-59b1-409c-b1cf-fd6b9c1a90b0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj" Oct 30 20:31:11.833616 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:11.833570 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktsl9\" (UniqueName: \"kubernetes.io/projected/18102e92-59b1-409c-b1cf-fd6b9c1a90b0-kube-api-access-ktsl9\") pod \"kuadrant-console-plugin-66c4fb575d-jwmvj\" (UID: \"18102e92-59b1-409c-b1cf-fd6b9c1a90b0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj" Oct 30 20:31:11.835469 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:11.835454 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6tg6g\"" Oct 30 20:31:11.838205 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:11.838191 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 20:31:11.934516 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:11.934486 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktsl9\" (UniqueName: \"kubernetes.io/projected/18102e92-59b1-409c-b1cf-fd6b9c1a90b0-kube-api-access-ktsl9\") pod \"kuadrant-console-plugin-66c4fb575d-jwmvj\" (UID: \"18102e92-59b1-409c-b1cf-fd6b9c1a90b0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj" Oct 30 20:31:11.934688 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:11.934528 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/18102e92-59b1-409c-b1cf-fd6b9c1a90b0-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jwmvj\" (UID: \"18102e92-59b1-409c-b1cf-fd6b9c1a90b0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj" Oct 30 20:31:11.934688 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:11.934550 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/18102e92-59b1-409c-b1cf-fd6b9c1a90b0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jwmvj\" (UID: \"18102e92-59b1-409c-b1cf-fd6b9c1a90b0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj" Oct 30 20:31:11.935258 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:11.935226 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/18102e92-59b1-409c-b1cf-fd6b9c1a90b0-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jwmvj\" (UID: \"18102e92-59b1-409c-b1cf-fd6b9c1a90b0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj" Oct 30 20:31:11.937026 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:11.937010 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/18102e92-59b1-409c-b1cf-fd6b9c1a90b0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jwmvj\" (UID: \"18102e92-59b1-409c-b1cf-fd6b9c1a90b0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj" Oct 30 20:31:11.993823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:11.993756 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktsl9\" (UniqueName: \"kubernetes.io/projected/18102e92-59b1-409c-b1cf-fd6b9c1a90b0-kube-api-access-ktsl9\") pod \"kuadrant-console-plugin-66c4fb575d-jwmvj\" (UID: \"18102e92-59b1-409c-b1cf-fd6b9c1a90b0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj" Oct 30 20:31:12.043633 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:12.043607 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj" Oct 30 20:31:12.415646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:12.415568 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj"] Oct 30 20:31:12.418400 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:31:12.418355 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18102e92_59b1_409c_b1cf_fd6b9c1a90b0.slice/crio-a06099ba62cef950772775dd19c3e861671f557cd7acef7d272c0178f27d2a41 WatchSource:0}: Error finding container a06099ba62cef950772775dd19c3e861671f557cd7acef7d272c0178f27d2a41: Status 404 returned error can't find the container with id a06099ba62cef950772775dd19c3e861671f557cd7acef7d272c0178f27d2a41 Oct 30 20:31:13.358570 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:13.358537 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj" event={"ID":"18102e92-59b1-409c-b1cf-fd6b9c1a90b0","Type":"ContainerStarted","Data":"a06099ba62cef950772775dd19c3e861671f557cd7acef7d272c0178f27d2a41"} Oct 30 20:31:15.204743 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:15.204711 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-rxdzr"] Oct 30 20:31:15.215773 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:15.210611 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rxdzr" Oct 30 20:31:15.267493 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:15.267202 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c463b05-a1b8-4ec7-b580-17934890bac3-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-rxdzr\" (UID: \"5c463b05-a1b8-4ec7-b580-17934890bac3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rxdzr" Oct 30 20:31:15.267493 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:15.267261 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdp2h\" (UniqueName: \"kubernetes.io/projected/5c463b05-a1b8-4ec7-b580-17934890bac3-kube-api-access-mdp2h\") pod \"cert-manager-cainjector-5c57bbd4fc-rxdzr\" (UID: \"5c463b05-a1b8-4ec7-b580-17934890bac3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rxdzr" Oct 30 20:31:15.267709 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:15.267580 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-rxdzr"] Oct 30 20:31:15.315051 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:15.315016 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xn6c9\"" Oct 30 20:31:15.368747 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:15.368044 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c463b05-a1b8-4ec7-b580-17934890bac3-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-rxdzr\" (UID: \"5c463b05-a1b8-4ec7-b580-17934890bac3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rxdzr" Oct 30 20:31:15.368747 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:15.368110 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdp2h\" (UniqueName: \"kubernetes.io/projected/5c463b05-a1b8-4ec7-b580-17934890bac3-kube-api-access-mdp2h\") pod \"cert-manager-cainjector-5c57bbd4fc-rxdzr\" (UID: \"5c463b05-a1b8-4ec7-b580-17934890bac3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rxdzr" Oct 30 20:31:15.442087 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:15.442021 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdp2h\" (UniqueName: \"kubernetes.io/projected/5c463b05-a1b8-4ec7-b580-17934890bac3-kube-api-access-mdp2h\") pod \"cert-manager-cainjector-5c57bbd4fc-rxdzr\" (UID: \"5c463b05-a1b8-4ec7-b580-17934890bac3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rxdzr" Oct 30 20:31:15.444143 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:15.444117 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c463b05-a1b8-4ec7-b580-17934890bac3-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-rxdzr\" (UID: \"5c463b05-a1b8-4ec7-b580-17934890bac3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rxdzr" Oct 30 20:31:15.530569 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:15.530520 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rxdzr" Oct 30 20:31:15.928039 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:15.926991 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-rxdzr"] Oct 30 20:31:15.936023 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:31:15.935994 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c463b05_a1b8_4ec7_b580_17934890bac3.slice/crio-a6bb09dd7627bae446913b15b5673d72030b7900da30fe3931079ca544a2751c WatchSource:0}: Error finding container a6bb09dd7627bae446913b15b5673d72030b7900da30fe3931079ca544a2751c: Status 404 returned error can't find the container with id a6bb09dd7627bae446913b15b5673d72030b7900da30fe3931079ca544a2751c Oct 30 20:31:16.381005 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:16.380938 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rxdzr" event={"ID":"5c463b05-a1b8-4ec7-b580-17934890bac3","Type":"ContainerStarted","Data":"a6bb09dd7627bae446913b15b5673d72030b7900da30fe3931079ca544a2751c"} Oct 30 20:31:17.385889 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.385863 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz"] Oct 30 20:31:17.468776 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.468717 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz"] Oct 30 20:31:17.468945 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.468846 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz" Oct 30 20:31:17.522940 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.522899 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 20:31:17.522940 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.522917 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 20:31:17.522940 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.522899 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 20:31:17.571881 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.571720 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 20:31:17.571881 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.571852 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ltg7l\"" Oct 30 20:31:17.587032 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.586991 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbaaf79a-ad29-4b7b-a914-4a2bae26f99e-tls-certs\") pod \"kube-auth-proxy-65fc6db558-kxhnz\" (UID: \"cbaaf79a-ad29-4b7b-a914-4a2bae26f99e\") " pod="openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz" Oct 30 20:31:17.587190 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.587061 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbaaf79a-ad29-4b7b-a914-4a2bae26f99e-tmp\") pod \"kube-auth-proxy-65fc6db558-kxhnz\" (UID: \"cbaaf79a-ad29-4b7b-a914-4a2bae26f99e\") " pod="openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz" Oct 30 20:31:17.587190 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.587088 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zftrw\" (UniqueName: \"kubernetes.io/projected/cbaaf79a-ad29-4b7b-a914-4a2bae26f99e-kube-api-access-zftrw\") pod \"kube-auth-proxy-65fc6db558-kxhnz\" (UID: \"cbaaf79a-ad29-4b7b-a914-4a2bae26f99e\") " pod="openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz" Oct 30 20:31:17.688792 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.688218 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbaaf79a-ad29-4b7b-a914-4a2bae26f99e-tls-certs\") pod \"kube-auth-proxy-65fc6db558-kxhnz\" (UID: \"cbaaf79a-ad29-4b7b-a914-4a2bae26f99e\") " pod="openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz" Oct 30 20:31:17.688792 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.688310 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbaaf79a-ad29-4b7b-a914-4a2bae26f99e-tmp\") pod \"kube-auth-proxy-65fc6db558-kxhnz\" (UID: \"cbaaf79a-ad29-4b7b-a914-4a2bae26f99e\") " pod="openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz" Oct 30 20:31:17.688792 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.688344 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftrw\" (UniqueName: \"kubernetes.io/projected/cbaaf79a-ad29-4b7b-a914-4a2bae26f99e-kube-api-access-zftrw\") pod \"kube-auth-proxy-65fc6db558-kxhnz\" (UID: \"cbaaf79a-ad29-4b7b-a914-4a2bae26f99e\") " pod="openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz" Oct 30 20:31:17.691355 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.691318 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbaaf79a-ad29-4b7b-a914-4a2bae26f99e-tmp\") pod \"kube-auth-proxy-65fc6db558-kxhnz\" (UID: \"cbaaf79a-ad29-4b7b-a914-4a2bae26f99e\") " pod="openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz" Oct 30 20:31:17.693232 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.693172 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbaaf79a-ad29-4b7b-a914-4a2bae26f99e-tls-certs\") pod \"kube-auth-proxy-65fc6db558-kxhnz\" (UID: \"cbaaf79a-ad29-4b7b-a914-4a2bae26f99e\") " pod="openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz" Oct 30 20:31:17.752166 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.752138 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zftrw\" (UniqueName: \"kubernetes.io/projected/cbaaf79a-ad29-4b7b-a914-4a2bae26f99e-kube-api-access-zftrw\") pod \"kube-auth-proxy-65fc6db558-kxhnz\" (UID: \"cbaaf79a-ad29-4b7b-a914-4a2bae26f99e\") " pod="openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz" Oct 30 20:31:17.812136 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:17.812032 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz" Oct 30 20:31:18.395206 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:18.395128 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pjpr9" event={"ID":"c90843ee-3a81-471e-ba63-4c3ef73eea83","Type":"ContainerStarted","Data":"bc28d6e3e99bc495620f321b6fdaea5cc2e23981bb75fb933c41b945cf793750"} Oct 30 20:31:18.395990 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:18.395959 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pjpr9" Oct 30 20:31:18.414153 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:18.414127 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz"] Oct 30 20:31:18.518807 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:18.518743 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pjpr9" podStartSLOduration=1.487551285 podStartE2EDuration="16.51872423s" totalImagesPullingTime="15.031172945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:18.517330549 +0000 UTC m=+241.443887464" watchObservedRunningTime="2026-10-30 20:31:18.51872423 +0000 UTC m=+241.445281148" Oct 30 20:31:18.599135 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:31:18.599096 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbaaf79a_ad29_4b7b_a914_4a2bae26f99e.slice/crio-250d987d3991790c5bd70b62f2f07218b47061b4769616dc5bab0058cf01aed5 WatchSource:0}: Error finding container 250d987d3991790c5bd70b62f2f07218b47061b4769616dc5bab0058cf01aed5: Status 404 returned error can't find the container with id 250d987d3991790c5bd70b62f2f07218b47061b4769616dc5bab0058cf01aed5 Oct 30 20:31:19.402037 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:19.401996 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz" event={"ID":"cbaaf79a-ad29-4b7b-a914-4a2bae26f99e","Type":"ContainerStarted","Data":"250d987d3991790c5bd70b62f2f07218b47061b4769616dc5bab0058cf01aed5"} Oct 30 20:31:19.404054 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:19.404028 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-f8bxl" event={"ID":"f2df5d57-9618-459b-b0fc-5a8fcfdde92e","Type":"ContainerStarted","Data":"6029d1e538bd55bfb88f805e40a4dd6929de06396e1d98bf9754ed65458f77ce"} Oct 30 20:31:19.526157 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:19.526105 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-f8bxl" podStartSLOduration=2.086234703 podStartE2EDuration="10.526092115s" totalImagesPullingTime="8.439857412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:19.524644728 +0000 UTC m=+242.451201655" watchObservedRunningTime="2026-10-30 20:31:19.526092115 +0000 UTC m=+242.452649031" Oct 30 20:31:20.412166 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:20.412101 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj" event={"ID":"18102e92-59b1-409c-b1cf-fd6b9c1a90b0","Type":"ContainerStarted","Data":"dfd4e73daeac03416dae26714195aa1e831ce62928fe4e1f517d7ae02545af8e"} Oct 30 20:31:20.414338 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:20.414299 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rxdzr" event={"ID":"5c463b05-a1b8-4ec7-b580-17934890bac3","Type":"ContainerStarted","Data":"19411f6e77be11f1d272ada1223cd5ecf9a7686bed0b06695713a3d6101dcef6"} Oct 30 20:31:20.414610 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:20.414359 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-f8bxl" Oct 30 20:31:20.530949 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:20.530897 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jwmvj" podStartSLOduration=1.741299154 podStartE2EDuration="9.530884318s" totalImagesPullingTime="7.789585164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:20.529946384 +0000 UTC m=+243.456503300" watchObservedRunningTime="2026-10-30 20:31:20.530884318 +0000 UTC m=+243.457441232" Oct 30 20:31:20.648679 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:20.648625 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rxdzr" podStartSLOduration=1.933768661 podStartE2EDuration="5.648610488s" totalImagesPullingTime="3.714841827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:20.647865724 +0000 UTC m=+243.574422640" watchObservedRunningTime="2026-10-30 20:31:20.648610488 +0000 UTC m=+243.575167403" Oct 30 20:31:22.425955 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:22.425921 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz" event={"ID":"cbaaf79a-ad29-4b7b-a914-4a2bae26f99e","Type":"ContainerStarted","Data":"5febddd67ff0f340cd683d83ae9b7b5808bb68103d63654848f690eeee3333ea"} Oct 30 20:31:22.542641 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:22.542589 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65fc6db558-kxhnz" podStartSLOduration=2.473342026 podStartE2EDuration="5.542575369s" totalImagesPullingTime="3.069233343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:22.54152801 +0000 UTC m=+245.468084937" watchObservedRunningTime="2026-10-30 20:31:22.542575369 +0000 UTC m=+245.469132283" Oct 30 20:31:26.421263 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:26.421233 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-f8bxl" Oct 30 20:31:29.621688 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:29.621650 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-flvs6"] Oct 30 20:31:29.624917 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:29.624892 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-flvs6" Oct 30 20:31:29.683732 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:29.683709 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-flvs6"] Oct 30 20:31:29.728483 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:29.728465 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7hp2l\"" Oct 30 20:31:29.788108 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:29.788083 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47ffe299-be14-4fb5-99c2-d3fea79bb132-bound-sa-token\") pod \"cert-manager-597b95cb84-flvs6\" (UID: \"47ffe299-be14-4fb5-99c2-d3fea79bb132\") " pod="cert-manager/cert-manager-597b95cb84-flvs6" Oct 30 20:31:29.788218 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:29.788124 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzq7k\" (UniqueName: \"kubernetes.io/projected/47ffe299-be14-4fb5-99c2-d3fea79bb132-kube-api-access-nzq7k\") pod \"cert-manager-597b95cb84-flvs6\" (UID: \"47ffe299-be14-4fb5-99c2-d3fea79bb132\") " pod="cert-manager/cert-manager-597b95cb84-flvs6" Oct 30 20:31:29.888496 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:29.888441 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47ffe299-be14-4fb5-99c2-d3fea79bb132-bound-sa-token\") pod \"cert-manager-597b95cb84-flvs6\" (UID: \"47ffe299-be14-4fb5-99c2-d3fea79bb132\") " pod="cert-manager/cert-manager-597b95cb84-flvs6" Oct 30 20:31:29.888496 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:29.888479 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzq7k\" (UniqueName: \"kubernetes.io/projected/47ffe299-be14-4fb5-99c2-d3fea79bb132-kube-api-access-nzq7k\") pod \"cert-manager-597b95cb84-flvs6\" (UID: \"47ffe299-be14-4fb5-99c2-d3fea79bb132\") " pod="cert-manager/cert-manager-597b95cb84-flvs6" Oct 30 20:31:29.947014 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:29.946981 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47ffe299-be14-4fb5-99c2-d3fea79bb132-bound-sa-token\") pod \"cert-manager-597b95cb84-flvs6\" (UID: \"47ffe299-be14-4fb5-99c2-d3fea79bb132\") " pod="cert-manager/cert-manager-597b95cb84-flvs6" Oct 30 20:31:29.947324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:29.947307 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzq7k\" (UniqueName: \"kubernetes.io/projected/47ffe299-be14-4fb5-99c2-d3fea79bb132-kube-api-access-nzq7k\") pod \"cert-manager-597b95cb84-flvs6\" (UID: \"47ffe299-be14-4fb5-99c2-d3fea79bb132\") " pod="cert-manager/cert-manager-597b95cb84-flvs6" Oct 30 20:31:30.236101 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:30.236077 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-flvs6" Oct 30 20:31:30.417121 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:30.417097 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pjpr9" Oct 30 20:31:30.649939 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:30.649898 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-flvs6"] Oct 30 20:31:30.652424 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:31:30.652398 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47ffe299_be14_4fb5_99c2_d3fea79bb132.slice/crio-bbc9fb674a0c4ef3c63400092b60c53fdae83bca0c6fc9b486374d25e708baec WatchSource:0}: Error finding container bbc9fb674a0c4ef3c63400092b60c53fdae83bca0c6fc9b486374d25e708baec: Status 404 returned error can't find the container with id bbc9fb674a0c4ef3c63400092b60c53fdae83bca0c6fc9b486374d25e708baec Oct 30 20:31:31.463311 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:31.463273 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-flvs6" event={"ID":"47ffe299-be14-4fb5-99c2-d3fea79bb132","Type":"ContainerStarted","Data":"42a17d5c287186abf1b834a892b88698b9f3adf5a7b819a100e44e11df1a3471"} Oct 30 20:31:31.463311 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:31.463314 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-flvs6" event={"ID":"47ffe299-be14-4fb5-99c2-d3fea79bb132","Type":"ContainerStarted","Data":"bbc9fb674a0c4ef3c63400092b60c53fdae83bca0c6fc9b486374d25e708baec"} Oct 30 20:31:31.590966 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:31.590915 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-flvs6" podStartSLOduration=2.5909015220000002 podStartE2EDuration="2.590901522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:31.589211778 +0000 UTC m=+254.515768693" watchObservedRunningTime="2026-10-30 20:31:31.590901522 +0000 UTC m=+254.517458438" Oct 30 20:31:41.254875 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:31:41.254836 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:31:51.349183 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:51.349148 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vqbw8"] Oct 30 20:31:51.377740 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:51.377707 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-vqbw8" Oct 30 20:31:51.408988 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:51.408961 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vqbw8"] Oct 30 20:31:51.457065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:51.457029 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ckfjh\"" Oct 30 20:31:51.481541 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:51.481509 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9lsj\" (UniqueName: \"kubernetes.io/projected/f7b2d1e7-f0c3-494c-a26a-cff5c6779b25-kube-api-access-c9lsj\") pod \"authorino-57f7f8659f-vqbw8\" (UID: \"f7b2d1e7-f0c3-494c-a26a-cff5c6779b25\") " pod="kuadrant-system/authorino-57f7f8659f-vqbw8" Oct 30 20:31:51.582213 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:51.582184 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9lsj\" (UniqueName: \"kubernetes.io/projected/f7b2d1e7-f0c3-494c-a26a-cff5c6779b25-kube-api-access-c9lsj\") pod \"authorino-57f7f8659f-vqbw8\" (UID: \"f7b2d1e7-f0c3-494c-a26a-cff5c6779b25\") " pod="kuadrant-system/authorino-57f7f8659f-vqbw8" Oct 30 20:31:51.642230 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:51.642159 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9lsj\" (UniqueName: \"kubernetes.io/projected/f7b2d1e7-f0c3-494c-a26a-cff5c6779b25-kube-api-access-c9lsj\") pod \"authorino-57f7f8659f-vqbw8\" (UID: \"f7b2d1e7-f0c3-494c-a26a-cff5c6779b25\") " pod="kuadrant-system/authorino-57f7f8659f-vqbw8" Oct 30 20:31:51.696299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:51.696271 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-vqbw8" Oct 30 20:31:52.091566 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:52.091539 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vqbw8"] Oct 30 20:31:52.094428 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:31:52.094380 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7b2d1e7_f0c3_494c_a26a_cff5c6779b25.slice/crio-49111af491cd061b48b9055f406e089450d152c399849cf10dba1a5e3af30353 WatchSource:0}: Error finding container 49111af491cd061b48b9055f406e089450d152c399849cf10dba1a5e3af30353: Status 404 returned error can't find the container with id 49111af491cd061b48b9055f406e089450d152c399849cf10dba1a5e3af30353 Oct 30 20:31:52.558220 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:52.558186 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-vqbw8" event={"ID":"f7b2d1e7-f0c3-494c-a26a-cff5c6779b25","Type":"ContainerStarted","Data":"49111af491cd061b48b9055f406e089450d152c399849cf10dba1a5e3af30353"} Oct 30 20:31:56.577734 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:56.577693 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-vqbw8" event={"ID":"f7b2d1e7-f0c3-494c-a26a-cff5c6779b25","Type":"ContainerStarted","Data":"ceaed389b63f3ed750378246e29e56897fd6d68d15cb9b542cbb8926a7285c03"} Oct 30 20:31:56.695514 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:56.695462 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-vqbw8" podStartSLOduration=1.9462321839999999 podStartE2EDuration="5.69544742s" totalImagesPullingTime="3.749215236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:56.694160155 +0000 UTC m=+279.620717089" watchObservedRunningTime="2026-10-30 20:31:56.69544742 +0000 UTC m=+279.622004335" Oct 30 20:31:57.375981 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:57.375946 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vqbw8"] Oct 30 20:31:58.586477 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:58.586436 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-vqbw8" podUID="f7b2d1e7-f0c3-494c-a26a-cff5c6779b25" containerName="authorino" containerID="cri-o://ceaed389b63f3ed750378246e29e56897fd6d68d15cb9b542cbb8926a7285c03" gracePeriod=30 Oct 30 20:31:58.827217 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:58.827191 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-vqbw8" Oct 30 20:31:58.948864 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:58.948790 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b2d1e7-f0c3-494c-a26a-cff5c6779b25-kube-api-access-c9lsj\" (UniqueName: \"kubernetes.io/projected/f7b2d1e7-f0c3-494c-a26a-cff5c6779b25-kube-api-access-c9lsj\") pod \"f7b2d1e7-f0c3-494c-a26a-cff5c6779b25\" (UID: \"f7b2d1e7-f0c3-494c-a26a-cff5c6779b25\") " Oct 30 20:31:58.951585 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:58.951551 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b2d1e7-f0c3-494c-a26a-cff5c6779b25-kube-api-access-c9lsj" pod "f7b2d1e7-f0c3-494c-a26a-cff5c6779b25" (UID: "f7b2d1e7-f0c3-494c-a26a-cff5c6779b25"). InnerVolumeSpecName "kube-api-access-c9lsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:59.050490 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:59.050450 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9lsj\" (UniqueName: \"kubernetes.io/projected/f7b2d1e7-f0c3-494c-a26a-cff5c6779b25-kube-api-access-c9lsj\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:59.592606 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:59.592569 5755 generic.go:356] "Generic (PLEG): container finished" podID="f7b2d1e7-f0c3-494c-a26a-cff5c6779b25" containerID="ceaed389b63f3ed750378246e29e56897fd6d68d15cb9b542cbb8926a7285c03" exitCode=0 Oct 30 20:31:59.593055 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:59.592617 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-vqbw8" Oct 30 20:31:59.593055 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:59.592656 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-vqbw8" event={"ID":"f7b2d1e7-f0c3-494c-a26a-cff5c6779b25","Type":"ContainerDied","Data":"ceaed389b63f3ed750378246e29e56897fd6d68d15cb9b542cbb8926a7285c03"} Oct 30 20:31:59.593055 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:59.592694 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-vqbw8" event={"ID":"f7b2d1e7-f0c3-494c-a26a-cff5c6779b25","Type":"ContainerDied","Data":"49111af491cd061b48b9055f406e089450d152c399849cf10dba1a5e3af30353"} Oct 30 20:31:59.593055 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:59.592710 5755 scope.go:122] "RemoveContainer" containerID="ceaed389b63f3ed750378246e29e56897fd6d68d15cb9b542cbb8926a7285c03" Oct 30 20:31:59.601067 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:59.601046 5755 scope.go:122] "RemoveContainer" containerID="ceaed389b63f3ed750378246e29e56897fd6d68d15cb9b542cbb8926a7285c03" Oct 30 20:31:59.601374 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:31:59.601355 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceaed389b63f3ed750378246e29e56897fd6d68d15cb9b542cbb8926a7285c03\": container with ID starting with ceaed389b63f3ed750378246e29e56897fd6d68d15cb9b542cbb8926a7285c03 not found: ID does not exist" containerID="ceaed389b63f3ed750378246e29e56897fd6d68d15cb9b542cbb8926a7285c03" Oct 30 20:31:59.601460 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:59.601408 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceaed389b63f3ed750378246e29e56897fd6d68d15cb9b542cbb8926a7285c03"} err="failed to get container status \"ceaed389b63f3ed750378246e29e56897fd6d68d15cb9b542cbb8926a7285c03\": rpc error: code = NotFound desc = could not find container \"ceaed389b63f3ed750378246e29e56897fd6d68d15cb9b542cbb8926a7285c03\": container with ID starting with ceaed389b63f3ed750378246e29e56897fd6d68d15cb9b542cbb8926a7285c03 not found: ID does not exist" Oct 30 20:31:59.757741 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:59.757706 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vqbw8"] Oct 30 20:31:59.807305 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:31:59.807267 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vqbw8"] Oct 30 20:32:01.209780 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:01.209748 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b2d1e7-f0c3-494c-a26a-cff5c6779b25" path="/var/lib/kubelet/pods/f7b2d1e7-f0c3-494c-a26a-cff5c6779b25/volumes" Oct 30 20:32:11.255218 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:32:11.255179 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:32:14.915539 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:14.915504 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-ltwv9"] Oct 30 20:32:14.915942 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:14.915839 5755 cpu_manager.go:414] "containerMap: removing container" podUID="f7b2d1e7-f0c3-494c-a26a-cff5c6779b25" containerName="authorino" Oct 30 20:32:14.915942 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:14.915850 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b2d1e7-f0c3-494c-a26a-cff5c6779b25" containerName="authorino" Oct 30 20:32:14.915942 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:14.915899 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b2d1e7-f0c3-494c-a26a-cff5c6779b25" containerName="authorino" Oct 30 20:32:14.918533 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:14.918513 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ltwv9" Oct 30 20:32:14.976862 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:14.976834 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 20:32:14.977824 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:14.977793 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-ltwv9"] Oct 30 20:32:14.991913 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:14.991885 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1f2f8b25-b9d5-400a-89d1-f553ad7273e6-data\") pod \"postgres-7cdbf6dc44-ltwv9\" (UID: \"1f2f8b25-b9d5-400a-89d1-f553ad7273e6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ltwv9" Oct 30 20:32:14.992049 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:14.991924 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbp2w\" (UniqueName: \"kubernetes.io/projected/1f2f8b25-b9d5-400a-89d1-f553ad7273e6-kube-api-access-wbp2w\") pod \"postgres-7cdbf6dc44-ltwv9\" (UID: \"1f2f8b25-b9d5-400a-89d1-f553ad7273e6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ltwv9" Oct 30 20:32:15.022923 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:15.022892 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:32:15.022923 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:15.022893 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:32:15.023087 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:15.022945 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-lj8s7\"" Oct 30 20:32:15.092915 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:15.092875 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1f2f8b25-b9d5-400a-89d1-f553ad7273e6-data\") pod \"postgres-7cdbf6dc44-ltwv9\" (UID: \"1f2f8b25-b9d5-400a-89d1-f553ad7273e6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ltwv9" Oct 30 20:32:15.093106 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:15.092927 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbp2w\" (UniqueName: \"kubernetes.io/projected/1f2f8b25-b9d5-400a-89d1-f553ad7273e6-kube-api-access-wbp2w\") pod \"postgres-7cdbf6dc44-ltwv9\" (UID: \"1f2f8b25-b9d5-400a-89d1-f553ad7273e6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ltwv9" Oct 30 20:32:15.093302 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:15.093281 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1f2f8b25-b9d5-400a-89d1-f553ad7273e6-data\") pod \"postgres-7cdbf6dc44-ltwv9\" (UID: \"1f2f8b25-b9d5-400a-89d1-f553ad7273e6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ltwv9" Oct 30 20:32:15.152190 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:15.152158 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbp2w\" (UniqueName: \"kubernetes.io/projected/1f2f8b25-b9d5-400a-89d1-f553ad7273e6-kube-api-access-wbp2w\") pod \"postgres-7cdbf6dc44-ltwv9\" (UID: \"1f2f8b25-b9d5-400a-89d1-f553ad7273e6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ltwv9" Oct 30 20:32:15.229971 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:15.229938 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ltwv9" Oct 30 20:32:15.603103 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:15.603041 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-ltwv9"] Oct 30 20:32:15.605732 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:32:15.605688 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2f8b25_b9d5_400a_89d1_f553ad7273e6.slice/crio-b287136d8b0e6fdfd374807eb547dd67f77446a17254d09a302bd5009230da7f WatchSource:0}: Error finding container b287136d8b0e6fdfd374807eb547dd67f77446a17254d09a302bd5009230da7f: Status 404 returned error can't find the container with id b287136d8b0e6fdfd374807eb547dd67f77446a17254d09a302bd5009230da7f Oct 30 20:32:15.666325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:15.666291 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ltwv9" event={"ID":"1f2f8b25-b9d5-400a-89d1-f553ad7273e6","Type":"ContainerStarted","Data":"b287136d8b0e6fdfd374807eb547dd67f77446a17254d09a302bd5009230da7f"} Oct 30 20:32:22.391102 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:22.391061 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 20:32:22.702830 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:22.702748 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ltwv9" event={"ID":"1f2f8b25-b9d5-400a-89d1-f553ad7273e6","Type":"ContainerStarted","Data":"210f4450930079d78c0ca5cc525db8fd79001e1120f50ec0bae481fe12f615c5"} Oct 30 20:32:22.702994 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:22.702931 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ltwv9" Oct 30 20:32:22.818324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:22.818276 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ltwv9" podStartSLOduration=2.085403021 podStartE2EDuration="8.818263356s" totalImagesPullingTime="6.732860335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:22.817251539 +0000 UTC m=+305.743808477" watchObservedRunningTime="2026-10-30 20:32:22.818263356 +0000 UTC m=+305.744820271" Oct 30 20:32:28.736761 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:28.736726 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ltwv9" Oct 30 20:32:41.255074 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:32:41.255037 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:32:44.541920 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.541887 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww"] Oct 30 20:32:44.545540 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.545523 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:32:44.603435 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.603410 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 20:32:44.630030 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.630005 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zb2ww\" (UID: \"0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:32:44.630145 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.630038 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4pj8\" (UniqueName: \"kubernetes.io/projected/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-kube-api-access-n4pj8\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zb2ww\" (UID: \"0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:32:44.630145 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.630112 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zb2ww\" (UID: \"0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:32:44.630230 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.630156 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zb2ww\" (UID: \"0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:32:44.653164 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.653129 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-xn26c\"" Oct 30 20:32:44.653272 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.653197 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 20:32:44.653350 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.653335 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 20:32:44.653436 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.653354 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 20:32:44.653436 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.653353 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 20:32:44.654797 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.654779 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww"] Oct 30 20:32:44.731182 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.731154 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zb2ww\" (UID: \"0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:32:44.731311 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.731188 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4pj8\" (UniqueName: \"kubernetes.io/projected/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-kube-api-access-n4pj8\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zb2ww\" (UID: \"0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:32:44.731311 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.731220 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zb2ww\" (UID: \"0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:32:44.731311 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.731271 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zb2ww\" (UID: \"0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:32:44.731523 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:32:44.731332 5755 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Oct 30 20:32:44.731523 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:32:44.731424 5755 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 30 20:32:44.731523 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:32:44.731430 5755 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-proxy-source podName:0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d nodeName:}" failed. No retries permitted until 2026-10-30 20:32:45.231412378 +0000 UTC m=+328.157969289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" (UID: "0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d") : configmap "usage-tenancy-proxy-source" not found Oct 30 20:32:44.731523 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:32:44.731496 5755 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-tls-cert podName:0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d nodeName:}" failed. No retries permitted until 2026-10-30 20:32:45.231477989 +0000 UTC m=+328.158034900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" (UID: "0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d") : secret "usage-logs-tenancy-proxy-tls" not found Oct 30 20:32:44.731927 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.731906 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zb2ww\" (UID: \"0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:32:44.795065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:44.795004 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4pj8\" (UniqueName: \"kubernetes.io/projected/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-kube-api-access-n4pj8\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zb2ww\" (UID: \"0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:32:45.235083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:45.235053 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zb2ww\" (UID: \"0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:32:45.235233 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:45.235106 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zb2ww\" (UID: \"0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:32:45.235885 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:45.235856 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zb2ww\" (UID: \"0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:32:45.237438 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:45.237416 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zb2ww\" (UID: \"0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:32:45.457899 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:45.457872 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:32:45.829921 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:45.829897 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww"] Oct 30 20:32:45.832087 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:32:45.832057 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1f2d0d_0c5f_45dc_bc49_5b5c112f095d.slice/crio-05fec4631a32ebcb95acbe398f615f6fb617a4fa3d87cd503ad7ec8b6a5b3c9e WatchSource:0}: Error finding container 05fec4631a32ebcb95acbe398f615f6fb617a4fa3d87cd503ad7ec8b6a5b3c9e: Status 404 returned error can't find the container with id 05fec4631a32ebcb95acbe398f615f6fb617a4fa3d87cd503ad7ec8b6a5b3c9e Oct 30 20:32:45.833807 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:45.833790 5755 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:32:46.812405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:46.812347 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" event={"ID":"0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d","Type":"ContainerStarted","Data":"05fec4631a32ebcb95acbe398f615f6fb617a4fa3d87cd503ad7ec8b6a5b3c9e"} Oct 30 20:32:55.748612 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.748577 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-7b5xp"] Oct 30 20:32:55.760628 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.760601 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-7b5xp" Oct 30 20:32:55.811061 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.811027 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-7b5xp"] Oct 30 20:32:55.813523 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.813469 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 20:32:55.813523 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.813469 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qzkqs\"" Oct 30 20:32:55.819993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.819969 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/054637a2-b34f-4384-b0af-72d53e95fc94-cert\") pod \"kserve-controller-manager-776cc66d47-7b5xp\" (UID: \"054637a2-b34f-4384-b0af-72d53e95fc94\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7b5xp" Oct 30 20:32:55.820149 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.820121 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sflb5\" (UniqueName: \"kubernetes.io/projected/054637a2-b34f-4384-b0af-72d53e95fc94-kube-api-access-sflb5\") pod \"kserve-controller-manager-776cc66d47-7b5xp\" (UID: \"054637a2-b34f-4384-b0af-72d53e95fc94\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7b5xp" Oct 30 20:32:55.860988 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.860960 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l"] Oct 30 20:32:55.865180 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.865156 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" Oct 30 20:32:55.920589 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.920543 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 20:32:55.920772 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.920663 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-r8p5t\"" Oct 30 20:32:55.920854 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.920783 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sflb5\" (UniqueName: \"kubernetes.io/projected/054637a2-b34f-4384-b0af-72d53e95fc94-kube-api-access-sflb5\") pod \"kserve-controller-manager-776cc66d47-7b5xp\" (UID: \"054637a2-b34f-4384-b0af-72d53e95fc94\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7b5xp" Oct 30 20:32:55.920854 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.920842 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/054637a2-b34f-4384-b0af-72d53e95fc94-cert\") pod \"kserve-controller-manager-776cc66d47-7b5xp\" (UID: \"054637a2-b34f-4384-b0af-72d53e95fc94\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7b5xp" Oct 30 20:32:55.920959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.920874 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85gjv\" (UniqueName: \"kubernetes.io/projected/23c02b52-e808-49cd-a4fb-8ead56e5f780-kube-api-access-85gjv\") pod \"llmisvc-controller-manager-869d5dcc5d-d6w8l\" (UID: \"23c02b52-e808-49cd-a4fb-8ead56e5f780\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" Oct 30 20:32:55.920959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.920944 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23c02b52-e808-49cd-a4fb-8ead56e5f780-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-d6w8l\" (UID: \"23c02b52-e808-49cd-a4fb-8ead56e5f780\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" Oct 30 20:32:55.921998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.921877 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l"] Oct 30 20:32:55.924034 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.924010 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/054637a2-b34f-4384-b0af-72d53e95fc94-cert\") pod \"kserve-controller-manager-776cc66d47-7b5xp\" (UID: \"054637a2-b34f-4384-b0af-72d53e95fc94\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7b5xp" Oct 30 20:32:55.977160 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.977115 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-fmbqh"] Oct 30 20:32:55.982294 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.982265 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sflb5\" (UniqueName: \"kubernetes.io/projected/054637a2-b34f-4384-b0af-72d53e95fc94-kube-api-access-sflb5\") pod \"kserve-controller-manager-776cc66d47-7b5xp\" (UID: \"054637a2-b34f-4384-b0af-72d53e95fc94\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7b5xp" Oct 30 20:32:55.982725 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:55.982707 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-fmbqh" Oct 30 20:32:56.022278 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.022212 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23c02b52-e808-49cd-a4fb-8ead56e5f780-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-d6w8l\" (UID: \"23c02b52-e808-49cd-a4fb-8ead56e5f780\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" Oct 30 20:32:56.022459 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.022302 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-668lh\" (UniqueName: \"kubernetes.io/projected/6d40b739-d433-4eb1-a1de-352637248047-kube-api-access-668lh\") pod \"model-serving-api-75bd89fdfc-fmbqh\" (UID: \"6d40b739-d433-4eb1-a1de-352637248047\") " pod="opendatahub/model-serving-api-75bd89fdfc-fmbqh" Oct 30 20:32:56.022459 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.022346 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85gjv\" (UniqueName: \"kubernetes.io/projected/23c02b52-e808-49cd-a4fb-8ead56e5f780-kube-api-access-85gjv\") pod \"llmisvc-controller-manager-869d5dcc5d-d6w8l\" (UID: \"23c02b52-e808-49cd-a4fb-8ead56e5f780\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" Oct 30 20:32:56.022459 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.022433 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d40b739-d433-4eb1-a1de-352637248047-tls-certs\") pod \"model-serving-api-75bd89fdfc-fmbqh\" (UID: \"6d40b739-d433-4eb1-a1de-352637248047\") " pod="opendatahub/model-serving-api-75bd89fdfc-fmbqh" Oct 30 20:32:56.025177 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.025153 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23c02b52-e808-49cd-a4fb-8ead56e5f780-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-d6w8l\" (UID: \"23c02b52-e808-49cd-a4fb-8ead56e5f780\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" Oct 30 20:32:56.036925 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.036903 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 20:32:56.038737 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.038712 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-fmbqh"] Oct 30 20:32:56.038934 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.038901 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-kqlc5\"" Oct 30 20:32:56.075261 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.075235 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-7b5xp" Oct 30 20:32:56.084959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.084930 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85gjv\" (UniqueName: \"kubernetes.io/projected/23c02b52-e808-49cd-a4fb-8ead56e5f780-kube-api-access-85gjv\") pod \"llmisvc-controller-manager-869d5dcc5d-d6w8l\" (UID: \"23c02b52-e808-49cd-a4fb-8ead56e5f780\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" Oct 30 20:32:56.095468 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.093750 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-lwcgf"] Oct 30 20:32:56.097801 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.097782 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-lwcgf" Oct 30 20:32:56.122945 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.122919 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/831b546f-889d-4f7a-8bc3-acc64dc3ce83-cert\") pod \"odh-model-controller-769875db56-lwcgf\" (UID: \"831b546f-889d-4f7a-8bc3-acc64dc3ce83\") " pod="opendatahub/odh-model-controller-769875db56-lwcgf" Oct 30 20:32:56.123073 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.123044 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-668lh\" (UniqueName: \"kubernetes.io/projected/6d40b739-d433-4eb1-a1de-352637248047-kube-api-access-668lh\") pod \"model-serving-api-75bd89fdfc-fmbqh\" (UID: \"6d40b739-d433-4eb1-a1de-352637248047\") " pod="opendatahub/model-serving-api-75bd89fdfc-fmbqh" Oct 30 20:32:56.123147 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.123104 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/831b546f-889d-4f7a-8bc3-acc64dc3ce83-metrics-certs\") pod \"odh-model-controller-769875db56-lwcgf\" (UID: \"831b546f-889d-4f7a-8bc3-acc64dc3ce83\") " pod="opendatahub/odh-model-controller-769875db56-lwcgf" Oct 30 20:32:56.123147 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.123139 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d40b739-d433-4eb1-a1de-352637248047-tls-certs\") pod \"model-serving-api-75bd89fdfc-fmbqh\" (UID: \"6d40b739-d433-4eb1-a1de-352637248047\") " pod="opendatahub/model-serving-api-75bd89fdfc-fmbqh" Oct 30 20:32:56.123235 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.123212 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgdk7\" (UniqueName: \"kubernetes.io/projected/831b546f-889d-4f7a-8bc3-acc64dc3ce83-kube-api-access-jgdk7\") pod \"odh-model-controller-769875db56-lwcgf\" (UID: \"831b546f-889d-4f7a-8bc3-acc64dc3ce83\") " pod="opendatahub/odh-model-controller-769875db56-lwcgf" Oct 30 20:32:56.127059 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.127035 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d40b739-d433-4eb1-a1de-352637248047-tls-certs\") pod \"model-serving-api-75bd89fdfc-fmbqh\" (UID: \"6d40b739-d433-4eb1-a1de-352637248047\") " pod="opendatahub/model-serving-api-75bd89fdfc-fmbqh" Oct 30 20:32:56.161632 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.161599 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 20:32:56.161800 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.161733 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-zbpjj\"" Oct 30 20:32:56.161867 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.161830 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 20:32:56.165718 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.165694 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-lwcgf"] Oct 30 20:32:56.179121 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.179097 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" Oct 30 20:32:56.181917 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.181890 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-668lh\" (UniqueName: \"kubernetes.io/projected/6d40b739-d433-4eb1-a1de-352637248047-kube-api-access-668lh\") pod \"model-serving-api-75bd89fdfc-fmbqh\" (UID: \"6d40b739-d433-4eb1-a1de-352637248047\") " pod="opendatahub/model-serving-api-75bd89fdfc-fmbqh" Oct 30 20:32:56.225089 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.224576 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgdk7\" (UniqueName: \"kubernetes.io/projected/831b546f-889d-4f7a-8bc3-acc64dc3ce83-kube-api-access-jgdk7\") pod \"odh-model-controller-769875db56-lwcgf\" (UID: \"831b546f-889d-4f7a-8bc3-acc64dc3ce83\") " pod="opendatahub/odh-model-controller-769875db56-lwcgf" Oct 30 20:32:56.225089 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.224636 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/831b546f-889d-4f7a-8bc3-acc64dc3ce83-cert\") pod \"odh-model-controller-769875db56-lwcgf\" (UID: \"831b546f-889d-4f7a-8bc3-acc64dc3ce83\") " pod="opendatahub/odh-model-controller-769875db56-lwcgf" Oct 30 20:32:56.225089 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.224698 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/831b546f-889d-4f7a-8bc3-acc64dc3ce83-metrics-certs\") pod \"odh-model-controller-769875db56-lwcgf\" (UID: \"831b546f-889d-4f7a-8bc3-acc64dc3ce83\") " pod="opendatahub/odh-model-controller-769875db56-lwcgf" Oct 30 20:32:56.227699 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.227673 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/831b546f-889d-4f7a-8bc3-acc64dc3ce83-metrics-certs\") pod \"odh-model-controller-769875db56-lwcgf\" (UID: \"831b546f-889d-4f7a-8bc3-acc64dc3ce83\") " pod="opendatahub/odh-model-controller-769875db56-lwcgf" Oct 30 20:32:56.233734 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.232364 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/831b546f-889d-4f7a-8bc3-acc64dc3ce83-cert\") pod \"odh-model-controller-769875db56-lwcgf\" (UID: \"831b546f-889d-4f7a-8bc3-acc64dc3ce83\") " pod="opendatahub/odh-model-controller-769875db56-lwcgf" Oct 30 20:32:56.284223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.284151 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgdk7\" (UniqueName: \"kubernetes.io/projected/831b546f-889d-4f7a-8bc3-acc64dc3ce83-kube-api-access-jgdk7\") pod \"odh-model-controller-769875db56-lwcgf\" (UID: \"831b546f-889d-4f7a-8bc3-acc64dc3ce83\") " pod="opendatahub/odh-model-controller-769875db56-lwcgf" Oct 30 20:32:56.304864 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.304833 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-fmbqh" Oct 30 20:32:56.416918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.416874 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-lwcgf" Oct 30 20:32:56.466030 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.466001 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-7b5xp"] Oct 30 20:32:56.471549 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:32:56.471516 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054637a2_b34f_4384_b0af_72d53e95fc94.slice/crio-2ed6d3228dcc1f4a61a7e4f74a99fb6b5dc4ae0407195788108422ac00d71bb4 WatchSource:0}: Error finding container 2ed6d3228dcc1f4a61a7e4f74a99fb6b5dc4ae0407195788108422ac00d71bb4: Status 404 returned error can't find the container with id 2ed6d3228dcc1f4a61a7e4f74a99fb6b5dc4ae0407195788108422ac00d71bb4 Oct 30 20:32:56.554044 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.554015 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l"] Oct 30 20:32:56.556312 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:32:56.556285 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod23c02b52_e808_49cd_a4fb_8ead56e5f780.slice/crio-a0b7ae2c1556776346c99e1ec6bbd2ff5bfec79a2845e190897cf1769f745c3b WatchSource:0}: Error finding container a0b7ae2c1556776346c99e1ec6bbd2ff5bfec79a2845e190897cf1769f745c3b: Status 404 returned error can't find the container with id a0b7ae2c1556776346c99e1ec6bbd2ff5bfec79a2845e190897cf1769f745c3b Oct 30 20:32:56.685256 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.685104 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-fmbqh"] Oct 30 20:32:56.687915 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:32:56.687882 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d40b739_d433_4eb1_a1de_352637248047.slice/crio-07ad8cb1048d97470f550aac5e5d9e5ab90579ede10aa8244234e74373170de0 WatchSource:0}: Error finding container 07ad8cb1048d97470f550aac5e5d9e5ab90579ede10aa8244234e74373170de0: Status 404 returned error can't find the container with id 07ad8cb1048d97470f550aac5e5d9e5ab90579ede10aa8244234e74373170de0 Oct 30 20:32:56.793572 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.793532 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-lwcgf"] Oct 30 20:32:56.797438 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:32:56.797404 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831b546f_889d_4f7a_8bc3_acc64dc3ce83.slice/crio-656f5b25eb76c1014da2ffbf9e6d6447acbd76be6ab8f72408fae7778e0870bc WatchSource:0}: Error finding container 656f5b25eb76c1014da2ffbf9e6d6447acbd76be6ab8f72408fae7778e0870bc: Status 404 returned error can't find the container with id 656f5b25eb76c1014da2ffbf9e6d6447acbd76be6ab8f72408fae7778e0870bc Oct 30 20:32:56.869882 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.869804 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-lwcgf" event={"ID":"831b546f-889d-4f7a-8bc3-acc64dc3ce83","Type":"ContainerStarted","Data":"656f5b25eb76c1014da2ffbf9e6d6447acbd76be6ab8f72408fae7778e0870bc"} Oct 30 20:32:56.871253 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.871221 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-fmbqh" event={"ID":"6d40b739-d433-4eb1-a1de-352637248047","Type":"ContainerStarted","Data":"07ad8cb1048d97470f550aac5e5d9e5ab90579ede10aa8244234e74373170de0"} Oct 30 20:32:56.872611 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.872577 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" event={"ID":"23c02b52-e808-49cd-a4fb-8ead56e5f780","Type":"ContainerStarted","Data":"a0b7ae2c1556776346c99e1ec6bbd2ff5bfec79a2845e190897cf1769f745c3b"} Oct 30 20:32:56.873949 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:32:56.873890 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-7b5xp" event={"ID":"054637a2-b34f-4384-b0af-72d53e95fc94","Type":"ContainerStarted","Data":"2ed6d3228dcc1f4a61a7e4f74a99fb6b5dc4ae0407195788108422ac00d71bb4"} Oct 30 20:33:00.905557 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:00.905512 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-fmbqh" event={"ID":"6d40b739-d433-4eb1-a1de-352637248047","Type":"ContainerStarted","Data":"90a3a41332e03e7b8ce1397c3d5c4aaea74ad045368fbb647150ef5f58b56c82"} Oct 30 20:33:00.906001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:00.905710 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-fmbqh" Oct 30 20:33:00.908132 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:00.907723 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-7b5xp" event={"ID":"054637a2-b34f-4384-b0af-72d53e95fc94","Type":"ContainerStarted","Data":"e93a01f861cfe15b5d5933c53ce725d8c08f878ec12beb83768722b000a601c2"} Oct 30 20:33:00.908132 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:00.907795 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-7b5xp" Oct 30 20:33:01.022479 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:01.022353 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-fmbqh" podStartSLOduration=2.856703902 podStartE2EDuration="6.022334586s" totalImagesPullingTime="3.165630684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:01.02075997 +0000 UTC m=+343.947316909" watchObservedRunningTime="2026-10-30 20:33:01.022334586 +0000 UTC m=+343.948891502" Oct 30 20:33:01.138157 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:01.137932 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-7b5xp" podStartSLOduration=2.259454837 podStartE2EDuration="6.137912751s" totalImagesPullingTime="3.878457914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:01.137505049 +0000 UTC m=+344.064061987" watchObservedRunningTime="2026-10-30 20:33:01.137912751 +0000 UTC m=+344.064469666" Oct 30 20:33:01.914927 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:01.914881 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" event={"ID":"23c02b52-e808-49cd-a4fb-8ead56e5f780","Type":"ContainerStarted","Data":"3eb98a5c1ed9c3bd184a95cc03e67fcdcd4678d82f901e0d14b1f32383e17c58"} Oct 30 20:33:01.915417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:01.914957 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" Oct 30 20:33:01.916612 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:01.916586 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" event={"ID":"0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d","Type":"ContainerStarted","Data":"9b537caceb7beef3448b383f3abfe5b305fa2a7423c649743693b7be96e8824d"} Oct 30 20:33:01.916738 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:01.916659 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:33:01.918148 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:01.918126 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-lwcgf" event={"ID":"831b546f-889d-4f7a-8bc3-acc64dc3ce83","Type":"ContainerStarted","Data":"13e6c47686bf8bc62d94019e0dd1042aeb195db05fa2b06c5cadf57b2c17a4bd"} Oct 30 20:33:02.032991 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:02.032943 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" podStartSLOduration=2.8282862890000002 podStartE2EDuration="7.032929678s" totalImagesPullingTime="4.204643389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:02.032083998 +0000 UTC m=+344.958640929" watchObservedRunningTime="2026-10-30 20:33:02.032929678 +0000 UTC m=+344.959486626" Oct 30 20:33:02.146956 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:02.146905 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-lwcgf" podStartSLOduration=1.8769044529999999 podStartE2EDuration="6.146890941s" totalImagesPullingTime="4.269986488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:02.14591751 +0000 UTC m=+345.072474444" watchObservedRunningTime="2026-10-30 20:33:02.146890941 +0000 UTC m=+345.073447855" Oct 30 20:33:02.260732 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:02.260680 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" podStartSLOduration=2.392956826 podStartE2EDuration="18.260665853s" totalImagesPullingTime="15.867709027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:02.260440949 +0000 UTC m=+345.186997866" watchObservedRunningTime="2026-10-30 20:33:02.260665853 +0000 UTC m=+345.187222771" Oct 30 20:33:02.923602 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:02.923569 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-lwcgf" Oct 30 20:33:11.253937 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:33:11.253898 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:33:11.923032 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:11.923003 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-fmbqh" Oct 30 20:33:12.928922 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:12.928888 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zb2ww" Oct 30 20:33:13.930559 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:13.930528 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-lwcgf" Oct 30 20:33:17.236009 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:17.235980 5755 scope.go:122] "RemoveContainer" containerID="353f1b1f4cd3fc6e41ba880f4fa88877efb099ff32a6cf8ab1c293d2800d9786" Oct 30 20:33:31.923567 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:31.923491 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-7b5xp" Oct 30 20:33:32.925309 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:33:32.925279 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" Oct 30 20:33:41.253899 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:33:41.253856 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:34:11.237468 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:11.237432 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qhnmd"] Oct 30 20:34:11.239787 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:11.239761 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qhnmd" Oct 30 20:34:11.256025 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:34:11.255995 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:34:11.298954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:11.298929 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qhnmd"] Oct 30 20:34:11.370366 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:11.370335 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6hh4\" (UniqueName: \"kubernetes.io/projected/4267521c-c2c4-4b5e-b10c-b3087da247ac-kube-api-access-h6hh4\") pod \"llmisvc-controller-manager-6466c554d8-qhnmd\" (UID: \"4267521c-c2c4-4b5e-b10c-b3087da247ac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qhnmd" Oct 30 20:34:11.370579 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:11.370419 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4267521c-c2c4-4b5e-b10c-b3087da247ac-cert\") pod \"llmisvc-controller-manager-6466c554d8-qhnmd\" (UID: \"4267521c-c2c4-4b5e-b10c-b3087da247ac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qhnmd" Oct 30 20:34:11.471409 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:11.471354 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4267521c-c2c4-4b5e-b10c-b3087da247ac-cert\") pod \"llmisvc-controller-manager-6466c554d8-qhnmd\" (UID: \"4267521c-c2c4-4b5e-b10c-b3087da247ac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qhnmd" Oct 30 20:34:11.471579 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:11.471477 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6hh4\" (UniqueName: \"kubernetes.io/projected/4267521c-c2c4-4b5e-b10c-b3087da247ac-kube-api-access-h6hh4\") pod \"llmisvc-controller-manager-6466c554d8-qhnmd\" (UID: \"4267521c-c2c4-4b5e-b10c-b3087da247ac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qhnmd" Oct 30 20:34:11.473882 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:11.473852 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4267521c-c2c4-4b5e-b10c-b3087da247ac-cert\") pod \"llmisvc-controller-manager-6466c554d8-qhnmd\" (UID: \"4267521c-c2c4-4b5e-b10c-b3087da247ac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qhnmd" Oct 30 20:34:11.529974 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:11.529902 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6hh4\" (UniqueName: \"kubernetes.io/projected/4267521c-c2c4-4b5e-b10c-b3087da247ac-kube-api-access-h6hh4\") pod \"llmisvc-controller-manager-6466c554d8-qhnmd\" (UID: \"4267521c-c2c4-4b5e-b10c-b3087da247ac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qhnmd" Oct 30 20:34:11.552773 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:11.552734 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qhnmd" Oct 30 20:34:11.923510 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:11.923173 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qhnmd"] Oct 30 20:34:11.928754 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:34:11.928722 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4267521c_c2c4_4b5e_b10c_b3087da247ac.slice/crio-38e3d5f571e37215ba9153e3e73ae4d3e75436e287777861c495c2928f018481 WatchSource:0}: Error finding container 38e3d5f571e37215ba9153e3e73ae4d3e75436e287777861c495c2928f018481: Status 404 returned error can't find the container with id 38e3d5f571e37215ba9153e3e73ae4d3e75436e287777861c495c2928f018481 Oct 30 20:34:12.249788 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:12.249757 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qhnmd" event={"ID":"4267521c-c2c4-4b5e-b10c-b3087da247ac","Type":"ContainerStarted","Data":"38e3d5f571e37215ba9153e3e73ae4d3e75436e287777861c495c2928f018481"} Oct 30 20:34:13.255628 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:13.255591 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qhnmd" event={"ID":"4267521c-c2c4-4b5e-b10c-b3087da247ac","Type":"ContainerStarted","Data":"abf3ac6dbe3f661766ff7d805a959eea6f04d21181044e204836532acef13af8"} Oct 30 20:34:13.256078 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:13.255732 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qhnmd" Oct 30 20:34:13.371299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:13.371239 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qhnmd" podStartSLOduration=1.6101555269999999 podStartE2EDuration="2.371224884s" totalImagesPullingTime="761.069357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:13.370302228 +0000 UTC m=+416.296859159" watchObservedRunningTime="2026-10-30 20:34:13.371224884 +0000 UTC m=+416.297781799" Oct 30 20:34:41.254731 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:34:41.254693 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:34:44.262405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:44.262368 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qhnmd" Oct 30 20:34:44.429083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:44.429053 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l"] Oct 30 20:34:44.429283 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:44.429261 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" podUID="23c02b52-e808-49cd-a4fb-8ead56e5f780" containerName="manager" containerID="cri-o://3eb98a5c1ed9c3bd184a95cc03e67fcdcd4678d82f901e0d14b1f32383e17c58" gracePeriod=30 Oct 30 20:34:44.671704 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:44.671677 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" Oct 30 20:34:44.760021 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:44.759988 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23c02b52-e808-49cd-a4fb-8ead56e5f780-cert\" (UniqueName: \"kubernetes.io/secret/23c02b52-e808-49cd-a4fb-8ead56e5f780-cert\") pod \"23c02b52-e808-49cd-a4fb-8ead56e5f780\" (UID: \"23c02b52-e808-49cd-a4fb-8ead56e5f780\") " Oct 30 20:34:44.760183 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:44.760028 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c02b52-e808-49cd-a4fb-8ead56e5f780-kube-api-access-85gjv\" (UniqueName: \"kubernetes.io/projected/23c02b52-e808-49cd-a4fb-8ead56e5f780-kube-api-access-85gjv\") pod \"23c02b52-e808-49cd-a4fb-8ead56e5f780\" (UID: \"23c02b52-e808-49cd-a4fb-8ead56e5f780\") " Oct 30 20:34:44.762334 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:44.762301 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23c02b52-e808-49cd-a4fb-8ead56e5f780-cert" pod "23c02b52-e808-49cd-a4fb-8ead56e5f780" (UID: "23c02b52-e808-49cd-a4fb-8ead56e5f780"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:34:44.762334 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:44.762318 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c02b52-e808-49cd-a4fb-8ead56e5f780-kube-api-access-85gjv" pod "23c02b52-e808-49cd-a4fb-8ead56e5f780" (UID: "23c02b52-e808-49cd-a4fb-8ead56e5f780"). InnerVolumeSpecName "kube-api-access-85gjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:44.860952 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:44.860874 5755 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23c02b52-e808-49cd-a4fb-8ead56e5f780-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:34:44.860952 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:44.860903 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85gjv\" (UniqueName: \"kubernetes.io/projected/23c02b52-e808-49cd-a4fb-8ead56e5f780-kube-api-access-85gjv\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:34:45.410889 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:45.410852 5755 generic.go:356] "Generic (PLEG): container finished" podID="23c02b52-e808-49cd-a4fb-8ead56e5f780" containerID="3eb98a5c1ed9c3bd184a95cc03e67fcdcd4678d82f901e0d14b1f32383e17c58" exitCode=0 Oct 30 20:34:45.411329 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:45.410909 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" Oct 30 20:34:45.411329 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:45.410907 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" event={"ID":"23c02b52-e808-49cd-a4fb-8ead56e5f780","Type":"ContainerDied","Data":"3eb98a5c1ed9c3bd184a95cc03e67fcdcd4678d82f901e0d14b1f32383e17c58"} Oct 30 20:34:45.411329 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:45.411067 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l" event={"ID":"23c02b52-e808-49cd-a4fb-8ead56e5f780","Type":"ContainerDied","Data":"a0b7ae2c1556776346c99e1ec6bbd2ff5bfec79a2845e190897cf1769f745c3b"} Oct 30 20:34:45.411329 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:45.411105 5755 scope.go:122] "RemoveContainer" containerID="3eb98a5c1ed9c3bd184a95cc03e67fcdcd4678d82f901e0d14b1f32383e17c58" Oct 30 20:34:45.419541 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:45.419522 5755 scope.go:122] "RemoveContainer" containerID="3eb98a5c1ed9c3bd184a95cc03e67fcdcd4678d82f901e0d14b1f32383e17c58" Oct 30 20:34:45.419792 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:34:45.419775 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb98a5c1ed9c3bd184a95cc03e67fcdcd4678d82f901e0d14b1f32383e17c58\": container with ID starting with 3eb98a5c1ed9c3bd184a95cc03e67fcdcd4678d82f901e0d14b1f32383e17c58 not found: ID does not exist" containerID="3eb98a5c1ed9c3bd184a95cc03e67fcdcd4678d82f901e0d14b1f32383e17c58" Oct 30 20:34:45.419869 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:45.419798 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb98a5c1ed9c3bd184a95cc03e67fcdcd4678d82f901e0d14b1f32383e17c58"} err="failed to get container status \"3eb98a5c1ed9c3bd184a95cc03e67fcdcd4678d82f901e0d14b1f32383e17c58\": rpc error: code = NotFound desc = could not find container \"3eb98a5c1ed9c3bd184a95cc03e67fcdcd4678d82f901e0d14b1f32383e17c58\": container with ID starting with 3eb98a5c1ed9c3bd184a95cc03e67fcdcd4678d82f901e0d14b1f32383e17c58 not found: ID does not exist" Oct 30 20:34:45.585562 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:45.585510 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l"] Oct 30 20:34:45.634971 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:45.634931 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d6w8l"] Oct 30 20:34:47.210929 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:34:47.210896 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c02b52-e808-49cd-a4fb-8ead56e5f780" path="/var/lib/kubelet/pods/23c02b52-e808-49cd-a4fb-8ead56e5f780/volumes" Oct 30 20:35:11.253799 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:35:11.253722 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:35:41.253958 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:35:41.253921 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:35:45.771171 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.771138 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624"] Oct 30 20:35:45.771564 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.771543 5755 cpu_manager.go:414] "containerMap: removing container" podUID="23c02b52-e808-49cd-a4fb-8ead56e5f780" containerName="manager" Oct 30 20:35:45.771564 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.771556 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c02b52-e808-49cd-a4fb-8ead56e5f780" containerName="manager" Oct 30 20:35:45.771644 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.771633 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c02b52-e808-49cd-a4fb-8ead56e5f780" containerName="manager" Oct 30 20:35:45.773609 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.773594 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.827082 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.827061 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 20:35:45.875992 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.875967 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hsn45\"" Oct 30 20:35:45.878235 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.878207 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624"] Oct 30 20:35:45.882151 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.882125 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.882275 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.882159 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d3f974-97c8-4aea-bc66-563950d17eb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.882275 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.882180 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.882275 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.882245 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.882437 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.882312 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.882437 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.882417 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.882502 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.882448 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.882502 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.882475 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch9pw\" (UniqueName: \"kubernetes.io/projected/26d3f974-97c8-4aea-bc66-563950d17eb9-kube-api-access-ch9pw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.882575 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.882555 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.983645 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.983617 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.983832 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.983656 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.983832 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.983674 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.983832 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.983693 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.983832 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.983720 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch9pw\" (UniqueName: \"kubernetes.io/projected/26d3f974-97c8-4aea-bc66-563950d17eb9-kube-api-access-ch9pw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.983832 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.983793 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.983832 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.983824 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.984139 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.983850 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d3f974-97c8-4aea-bc66-563950d17eb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.984139 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.983884 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.984248 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.984165 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.984248 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.984181 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.984352 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.984280 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.984600 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.984569 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.984777 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.984755 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d3f974-97c8-4aea-bc66-563950d17eb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.986172 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.986147 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:45.986643 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:45.986621 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:46.044115 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:46.044054 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:46.044215 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:46.044109 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch9pw\" (UniqueName: \"kubernetes.io/projected/26d3f974-97c8-4aea-bc66-563950d17eb9-kube-api-access-ch9pw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fl624\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:46.084939 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:46.084910 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:46.513897 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:46.513866 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624"] Oct 30 20:35:46.516170 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:35:46.516137 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d3f974_97c8_4aea_bc66_563950d17eb9.slice/crio-92223e273ff3b02adc2e4f5e27fedbacc8e0369f228a2f952f63b28967575baf WatchSource:0}: Error finding container 92223e273ff3b02adc2e4f5e27fedbacc8e0369f228a2f952f63b28967575baf: Status 404 returned error can't find the container with id 92223e273ff3b02adc2e4f5e27fedbacc8e0369f228a2f952f63b28967575baf Oct 30 20:35:46.715565 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:46.715531 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" event={"ID":"26d3f974-97c8-4aea-bc66-563950d17eb9","Type":"ContainerStarted","Data":"92223e273ff3b02adc2e4f5e27fedbacc8e0369f228a2f952f63b28967575baf"} Oct 30 20:35:50.193795 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:50.193756 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:35:50.194215 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:50.193847 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:35:50.194215 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:50.193898 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:35:50.737964 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:50.737926 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" event={"ID":"26d3f974-97c8-4aea-bc66-563950d17eb9","Type":"ContainerStarted","Data":"a2eb4f452e5486da588d92ec624f2e23a97ed9b008b3a3d1fad0968d2addb12e"} Oct 30 20:35:50.963519 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:50.963457 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" podStartSLOduration=2.2899562270000002 podStartE2EDuration="5.963442752s" totalImagesPullingTime="3.673486525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:50.913547972 +0000 UTC m=+513.840104888" watchObservedRunningTime="2026-10-30 20:35:50.963442752 +0000 UTC m=+513.889999687" Oct 30 20:35:51.088879 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:51.088745 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:51.091164 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:51.090819 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fl624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:35:51.091164 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:51.090879 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" podUID="26d3f974-97c8-4aea-bc66-563950d17eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:35:52.086845 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:52.086804 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fl624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:35:52.087470 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:52.087427 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" podUID="26d3f974-97c8-4aea-bc66-563950d17eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:35:53.087870 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:53.087835 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fl624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:35:53.088498 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:53.088465 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" podUID="26d3f974-97c8-4aea-bc66-563950d17eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:35:54.088411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:54.087800 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fl624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:35:54.088411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:54.087862 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" podUID="26d3f974-97c8-4aea-bc66-563950d17eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:35:55.088787 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:55.088746 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fl624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:35:55.089368 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:55.089334 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" podUID="26d3f974-97c8-4aea-bc66-563950d17eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:35:56.085854 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:56.085819 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fl624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:35:56.086042 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:56.085873 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" podUID="26d3f974-97c8-4aea-bc66-563950d17eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:35:57.085873 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:57.085839 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fl624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:35:57.086272 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:57.085893 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" podUID="26d3f974-97c8-4aea-bc66-563950d17eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:35:58.326742 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:58.326709 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:58.327192 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:58.327098 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:35:58.327894 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:35:58.327870 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:36:11.254932 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:36:11.254889 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:36:41.253877 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:36:41.253836 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:37:11.255022 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:37:11.254979 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:37:28.495597 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:28.495552 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn"] Oct 30 20:37:28.498782 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:28.498765 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" Oct 30 20:37:28.593807 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:28.593772 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkbnt\" (UniqueName: \"kubernetes.io/projected/312ff01e-18e1-4d99-9dfd-67df3b8e640f-kube-api-access-bkbnt\") pod \"payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn\" (UID: \"312ff01e-18e1-4d99-9dfd-67df3b8e640f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" Oct 30 20:37:28.593965 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:28.593813 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312ff01e-18e1-4d99-9dfd-67df3b8e640f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn\" (UID: \"312ff01e-18e1-4d99-9dfd-67df3b8e640f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" Oct 30 20:37:28.601405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:28.601358 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-f6ebf5-dockercfg-6d8dh\"" Oct 30 20:37:28.601405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:28.601364 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-f6ebf5\"" Oct 30 20:37:28.603080 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:28.603061 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn"] Oct 30 20:37:28.694443 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:28.694409 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkbnt\" (UniqueName: \"kubernetes.io/projected/312ff01e-18e1-4d99-9dfd-67df3b8e640f-kube-api-access-bkbnt\") pod \"payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn\" (UID: \"312ff01e-18e1-4d99-9dfd-67df3b8e640f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" Oct 30 20:37:28.694443 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:28.694450 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312ff01e-18e1-4d99-9dfd-67df3b8e640f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn\" (UID: \"312ff01e-18e1-4d99-9dfd-67df3b8e640f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" Oct 30 20:37:28.695123 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:28.695102 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312ff01e-18e1-4d99-9dfd-67df3b8e640f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn\" (UID: \"312ff01e-18e1-4d99-9dfd-67df3b8e640f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" Oct 30 20:37:28.752436 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:28.752354 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkbnt\" (UniqueName: \"kubernetes.io/projected/312ff01e-18e1-4d99-9dfd-67df3b8e640f-kube-api-access-bkbnt\") pod \"payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn\" (UID: \"312ff01e-18e1-4d99-9dfd-67df3b8e640f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" Oct 30 20:37:28.811116 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:28.811073 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" Oct 30 20:37:29.181791 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:29.181699 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn"] Oct 30 20:37:29.184694 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:37:29.184666 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312ff01e_18e1_4d99_9dfd_67df3b8e640f.slice/crio-586e7281791a6ef86130372fe766ff1822c2f56b7e52db383facf2d9f22769a6 WatchSource:0}: Error finding container 586e7281791a6ef86130372fe766ff1822c2f56b7e52db383facf2d9f22769a6: Status 404 returned error can't find the container with id 586e7281791a6ef86130372fe766ff1822c2f56b7e52db383facf2d9f22769a6 Oct 30 20:37:29.210685 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:29.210656 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" event={"ID":"312ff01e-18e1-4d99-9dfd-67df3b8e640f","Type":"ContainerStarted","Data":"586e7281791a6ef86130372fe766ff1822c2f56b7e52db383facf2d9f22769a6"} Oct 30 20:37:32.225845 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:32.225814 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" event={"ID":"312ff01e-18e1-4d99-9dfd-67df3b8e640f","Type":"ContainerStarted","Data":"9a3703de5f02d5004b7709efc5cf47b6fd047f268008d977b1225b1229b02a39"} Oct 30 20:37:32.226209 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:32.225926 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" Oct 30 20:37:32.343119 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:32.343007 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" podStartSLOduration=1.524050739 podStartE2EDuration="4.342995199s" totalImagesPullingTime="2.81894446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:32.342586235 +0000 UTC m=+615.269143167" watchObservedRunningTime="2026-10-30 20:37:32.342995199 +0000 UTC m=+615.269552113" Oct 30 20:37:41.255474 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:37:41.255437 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:37:43.039687 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.039654 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq"] Oct 30 20:37:43.043397 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.043369 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:43.120310 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.120283 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbjdx\" (UniqueName: \"kubernetes.io/projected/50baec58-01fc-4d2a-914a-1baf6a6cda9c-kube-api-access-pbjdx\") pod \"maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:43.120472 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.120316 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:43.120472 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.120451 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50baec58-01fc-4d2a-914a-1baf6a6cda9c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:43.120604 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.120506 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-metrics-certs\") pod \"maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:43.145864 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.145835 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-386743-serving-cert\"" Oct 30 20:37:43.145982 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.145921 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rswnw\"" Oct 30 20:37:43.145982 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.145946 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 20:37:43.147594 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.147563 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq"] Oct 30 20:37:43.221526 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.221499 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-metrics-certs\") pod \"maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:43.221689 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.221572 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbjdx\" (UniqueName: \"kubernetes.io/projected/50baec58-01fc-4d2a-914a-1baf6a6cda9c-kube-api-access-pbjdx\") pod \"maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:43.221689 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.221591 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:43.221689 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.221626 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50baec58-01fc-4d2a-914a-1baf6a6cda9c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:43.221861 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:37:43.221726 5755 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-serving-cert: secret "maas-api-e2e-worker-w5-386743-serving-cert" not found Oct 30 20:37:43.221861 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:37:43.221796 5755 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-maas-api-tls podName:50baec58-01fc-4d2a-914a-1baf6a6cda9c nodeName:}" failed. No retries permitted until 2026-10-30 20:37:43.721775201 +0000 UTC m=+626.648332095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-maas-api-tls") pod "maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" (UID: "50baec58-01fc-4d2a-914a-1baf6a6cda9c") : secret "maas-api-e2e-worker-w5-386743-serving-cert" not found Oct 30 20:37:43.222234 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.222217 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50baec58-01fc-4d2a-914a-1baf6a6cda9c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:43.223971 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.223950 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-metrics-certs\") pod \"maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:43.233310 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.233291 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" Oct 30 20:37:43.281670 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.281645 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbjdx\" (UniqueName: \"kubernetes.io/projected/50baec58-01fc-4d2a-914a-1baf6a6cda9c-kube-api-access-pbjdx\") pod \"maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:43.725961 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.725929 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:43.728463 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.728435 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:43.956685 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:43.956652 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:44.377069 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:44.377043 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq"] Oct 30 20:37:44.379257 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:37:44.379233 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50baec58_01fc_4d2a_914a_1baf6a6cda9c.slice/crio-7595405adbb3e3ef4a17d0d7b7cdbc6719e875537915fc277784931588cda3dc WatchSource:0}: Error finding container 7595405adbb3e3ef4a17d0d7b7cdbc6719e875537915fc277784931588cda3dc: Status 404 returned error can't find the container with id 7595405adbb3e3ef4a17d0d7b7cdbc6719e875537915fc277784931588cda3dc Oct 30 20:37:45.285679 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:45.285641 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" event={"ID":"50baec58-01fc-4d2a-914a-1baf6a6cda9c","Type":"ContainerStarted","Data":"7595405adbb3e3ef4a17d0d7b7cdbc6719e875537915fc277784931588cda3dc"} Oct 30 20:37:48.304842 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:48.304801 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" event={"ID":"50baec58-01fc-4d2a-914a-1baf6a6cda9c","Type":"ContainerStarted","Data":"b4a0b97151c1a3a605f8f322c34259da88fafe99ba7d95db27bbbd4c28e6357e"} Oct 30 20:37:48.305221 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:48.304865 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:48.421745 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:48.421697 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" podStartSLOduration=3.438061825 podStartE2EDuration="6.421683987s" totalImagesPullingTime="2.983622162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:48.420084983 +0000 UTC m=+631.346641910" watchObservedRunningTime="2026-10-30 20:37:48.421683987 +0000 UTC m=+631.348240951" Oct 30 20:37:53.503256 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:53.503221 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42"] Oct 30 20:37:53.509377 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:53.509350 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:37:53.613066 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:53.613038 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-86d620-serving-cert\"" Oct 30 20:37:53.614836 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:53.614813 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42"] Oct 30 20:37:53.617525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:53.617498 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:37:53.617642 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:53.617625 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-metrics-certs\") pod \"maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:37:53.617700 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:53.617652 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l28bw\" (UniqueName: \"kubernetes.io/projected/85e81941-ca08-4234-9288-0dd4ecddf92f-kube-api-access-l28bw\") pod \"maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:37:53.617700 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:53.617693 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e81941-ca08-4234-9288-0dd4ecddf92f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:37:53.718355 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:53.718324 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:37:53.718530 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:53.718381 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-metrics-certs\") pod \"maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:37:53.718530 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:53.718420 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l28bw\" (UniqueName: \"kubernetes.io/projected/85e81941-ca08-4234-9288-0dd4ecddf92f-kube-api-access-l28bw\") pod \"maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:37:53.718530 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:53.718454 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e81941-ca08-4234-9288-0dd4ecddf92f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:37:53.718530 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:37:53.718486 5755 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-serving-cert: secret "maas-api-e2e-worker-w1-86d620-serving-cert" not found Oct 30 20:37:53.718685 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:37:53.718546 5755 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-maas-api-tls podName:85e81941-ca08-4234-9288-0dd4ecddf92f nodeName:}" failed. No retries permitted until 2026-10-30 20:37:54.218531371 +0000 UTC m=+637.145088268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-maas-api-tls") pod "maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" (UID: "85e81941-ca08-4234-9288-0dd4ecddf92f") : secret "maas-api-e2e-worker-w1-86d620-serving-cert" not found Oct 30 20:37:53.719131 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:53.719108 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e81941-ca08-4234-9288-0dd4ecddf92f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:37:53.720875 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:53.720854 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-metrics-certs\") pod \"maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:37:53.779479 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:53.779372 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l28bw\" (UniqueName: \"kubernetes.io/projected/85e81941-ca08-4234-9288-0dd4ecddf92f-kube-api-access-l28bw\") pod \"maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:37:54.223550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:54.223508 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:37:54.226008 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:54.225987 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:37:54.315139 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:54.315115 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 20:37:54.421793 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:54.421761 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:37:54.805799 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:54.805773 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42"] Oct 30 20:37:54.807800 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:37:54.807772 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e81941_ca08_4234_9288_0dd4ecddf92f.slice/crio-8517b1ec09b22e4f6b3fa8bc7fda7837adc1cec6f15f8f0364512bad254976fe WatchSource:0}: Error finding container 8517b1ec09b22e4f6b3fa8bc7fda7837adc1cec6f15f8f0364512bad254976fe: Status 404 returned error can't find the container with id 8517b1ec09b22e4f6b3fa8bc7fda7837adc1cec6f15f8f0364512bad254976fe Oct 30 20:37:54.809061 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:54.809039 5755 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:37:55.339249 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:55.339213 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" event={"ID":"85e81941-ca08-4234-9288-0dd4ecddf92f","Type":"ContainerStarted","Data":"8517b1ec09b22e4f6b3fa8bc7fda7837adc1cec6f15f8f0364512bad254976fe"} Oct 30 20:37:56.345691 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:56.345654 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" event={"ID":"85e81941-ca08-4234-9288-0dd4ecddf92f","Type":"ContainerStarted","Data":"b645c249272cb0b0ee9b701cae19ccdace31539dec6c266f8192d3786391d2cc"} Oct 30 20:37:56.346104 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:56.345775 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:37:56.478693 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:37:56.478616 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" podStartSLOduration=2.7017632430000003 podStartE2EDuration="3.478599101s" totalImagesPullingTime="776.835858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:56.478464911 +0000 UTC m=+639.405021828" watchObservedRunningTime="2026-10-30 20:37:56.478599101 +0000 UTC m=+639.405156016" Oct 30 20:38:02.355996 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:02.355967 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:38:09.621776 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.621744 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f"] Oct 30 20:38:09.625338 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.625323 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.699922 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.699894 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f"] Oct 30 20:38:09.731222 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.731196 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ee3fad\"/\"kube-root-ca.crt\"" Oct 30 20:38:09.731368 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.731314 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ee3fad\"/\"openshift-service-ca.crt\"" Oct 30 20:38:09.731368 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.731345 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ee3fad\"/\"default-dockercfg-gjcdt\"" Oct 30 20:38:09.731460 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.731373 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ee3fad\"/\"simulator-w6-ee3fad-kserve-self-signed-certs\"" Oct 30 20:38:09.765249 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.765223 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-tmp-dir\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.765445 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.765253 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxkx5\" (UniqueName: \"kubernetes.io/projected/aa094251-9344-47b6-bbcf-79823b693371-kube-api-access-dxkx5\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.765445 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.765273 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-dshm\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.765445 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.765292 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-model-cache\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.765587 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.765559 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-home\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.765633 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.765593 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa094251-9344-47b6-bbcf-79823b693371-tls-certs\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.866930 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.866902 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-home\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.866930 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.866930 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa094251-9344-47b6-bbcf-79823b693371-tls-certs\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.867170 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.866975 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-tmp-dir\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.867170 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.866991 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxkx5\" (UniqueName: \"kubernetes.io/projected/aa094251-9344-47b6-bbcf-79823b693371-kube-api-access-dxkx5\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.867170 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.867016 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-dshm\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.867170 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.867048 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-model-cache\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.867456 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.867431 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-home\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.867539 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.867464 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-tmp-dir\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.867684 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.867667 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-model-cache\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.869469 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.869448 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-dshm\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.869707 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.869691 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa094251-9344-47b6-bbcf-79823b693371-tls-certs\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.925974 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.925909 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxkx5\" (UniqueName: \"kubernetes.io/projected/aa094251-9344-47b6-bbcf-79823b693371-kube-api-access-dxkx5\") pod \"simulator-w6-ee3fad-kserve-77c46d8787-mz55f\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:09.936708 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:09.936683 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:10.313128 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:10.313104 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f"] Oct 30 20:38:10.314902 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:38:10.314878 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa094251_9344_47b6_bbcf_79823b693371.slice/crio-ea166dc46deb35a43ece39119583ecd16caf41f8c2c6f404fe6aab840490d16b WatchSource:0}: Error finding container ea166dc46deb35a43ece39119583ecd16caf41f8c2c6f404fe6aab840490d16b: Status 404 returned error can't find the container with id ea166dc46deb35a43ece39119583ecd16caf41f8c2c6f404fe6aab840490d16b Oct 30 20:38:10.413140 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:10.413100 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" event={"ID":"aa094251-9344-47b6-bbcf-79823b693371","Type":"ContainerStarted","Data":"ea166dc46deb35a43ece39119583ecd16caf41f8c2c6f404fe6aab840490d16b"} Oct 30 20:38:11.262811 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:38:11.262768 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:38:12.426312 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:12.426279 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" event={"ID":"aa094251-9344-47b6-bbcf-79823b693371","Type":"ContainerStarted","Data":"4368d885a4d2e5ac0f5bb4a7ea6f14a2ab4962f9682d3ccd8d17d818a14eb91d"} Oct 30 20:38:12.545022 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:12.544968 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podStartSLOduration=2.025216778 podStartE2EDuration="3.544953887s" totalImagesPullingTime="1.519737109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:12.543787851 +0000 UTC m=+655.470344766" watchObservedRunningTime="2026-10-30 20:38:12.544953887 +0000 UTC m=+655.471510802" Oct 30 20:38:12.937542 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:12.937510 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:19.937620 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:19.937586 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:19.950420 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:19.950380 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:20.476901 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:20.476874 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:38:31.801256 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.801218 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq"] Oct 30 20:38:31.804817 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.804801 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.858300 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.858273 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f6ebf5\"/\"default-dockercfg-ffzrb\"" Oct 30 20:38:31.861607 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.861578 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197b515b-ee40-4966-aa7f-f20e16001039-tls-certs\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.861747 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.861618 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-tmp-dir\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.861747 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.861711 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-home\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.861840 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.861754 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwvq\" (UniqueName: \"kubernetes.io/projected/197b515b-ee40-4966-aa7f-f20e16001039-kube-api-access-qzwvq\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.861913 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.861881 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-model-cache\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.862074 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.861938 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-dshm\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.906954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.906930 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f6ebf5\"/\"simulator-w3-f6ebf5-kserve-self-signed-certs\"" Oct 30 20:38:31.907080 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.906991 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f6ebf5\"/\"kube-root-ca.crt\"" Oct 30 20:38:31.907080 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.907003 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f6ebf5\"/\"openshift-service-ca.crt\"" Oct 30 20:38:31.909162 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.909143 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq"] Oct 30 20:38:31.963272 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.963237 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-model-cache\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.963479 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.963287 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-dshm\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.963479 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.963329 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197b515b-ee40-4966-aa7f-f20e16001039-tls-certs\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.963479 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.963352 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-tmp-dir\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.963479 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.963417 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-home\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.963479 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.963439 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwvq\" (UniqueName: \"kubernetes.io/projected/197b515b-ee40-4966-aa7f-f20e16001039-kube-api-access-qzwvq\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.963772 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.963683 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-model-cache\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.963772 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.963740 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-tmp-dir\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.963855 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.963811 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-home\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.965744 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.965725 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-dshm\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:31.966010 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:31.965989 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197b515b-ee40-4966-aa7f-f20e16001039-tls-certs\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:32.021993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:32.021955 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwvq\" (UniqueName: \"kubernetes.io/projected/197b515b-ee40-4966-aa7f-f20e16001039-kube-api-access-qzwvq\") pod \"simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:32.116561 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:32.116476 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:32.496043 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:32.496019 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq"] Oct 30 20:38:32.497873 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:38:32.497836 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod197b515b_ee40_4966_aa7f_f20e16001039.slice/crio-b2ae8c98bac79c0c08a530578ea0be9b6d852446e6c137982311e10439b8e76e WatchSource:0}: Error finding container b2ae8c98bac79c0c08a530578ea0be9b6d852446e6c137982311e10439b8e76e: Status 404 returned error can't find the container with id b2ae8c98bac79c0c08a530578ea0be9b6d852446e6c137982311e10439b8e76e Oct 30 20:38:32.528452 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:32.528424 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" event={"ID":"197b515b-ee40-4966-aa7f-f20e16001039","Type":"ContainerStarted","Data":"b2ae8c98bac79c0c08a530578ea0be9b6d852446e6c137982311e10439b8e76e"} Oct 30 20:38:33.535349 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:33.535315 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" event={"ID":"197b515b-ee40-4966-aa7f-f20e16001039","Type":"ContainerStarted","Data":"949e13b714e0079aa32fd5e3ab1a9bd89db7d2453b132a8988022f896bb84af5"} Oct 30 20:38:33.651983 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:33.651937 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podStartSLOduration=2.6519241989999998 podStartE2EDuration="2.651924199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:33.651478673 +0000 UTC m=+676.578035589" watchObservedRunningTime="2026-10-30 20:38:33.651924199 +0000 UTC m=+676.578481110" Oct 30 20:38:34.117135 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:34.117102 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:41.254216 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:38:41.254177 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:38:42.116911 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:42.116875 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:42.129178 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:42.129152 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:38:42.588815 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:38:42.588789 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:39:03.268436 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.268400 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk"] Oct 30 20:39:03.272472 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.272442 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.327021 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.326991 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk"] Oct 30 20:39:03.361896 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.361864 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-tmp-dir\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.361896 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.361897 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-model-cache\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.362148 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.361916 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-dshm\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.362148 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.361933 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tszx2\" (UniqueName: \"kubernetes.io/projected/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-kube-api-access-tszx2\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.362148 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.362092 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-tls-certs\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.362286 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.362158 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-home\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.374262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.374236 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-386743\"/\"kube-root-ca.crt\"" Oct 30 20:39:03.374499 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.374481 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-386743\"/\"simulator-w5-386743-kserve-self-signed-certs\"" Oct 30 20:39:03.375193 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.375179 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-386743\"/\"openshift-service-ca.crt\"" Oct 30 20:39:03.375445 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.375432 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-386743\"/\"default-dockercfg-7mljt\"" Oct 30 20:39:03.463348 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.463313 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-home\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.463538 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.463377 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-tmp-dir\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.463538 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.463424 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-model-cache\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.463538 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.463438 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-dshm\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.463538 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.463453 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tszx2\" (UniqueName: \"kubernetes.io/projected/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-kube-api-access-tszx2\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.463538 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.463473 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-tls-certs\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.463796 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.463775 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-home\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.463844 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.463811 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-tmp-dir\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.463887 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.463870 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-model-cache\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.466103 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.466074 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-dshm\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.466268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.466136 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-tls-certs\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.522024 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.521958 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tszx2\" (UniqueName: \"kubernetes.io/projected/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-kube-api-access-tszx2\") pod \"simulator-w5-386743-kserve-55747d678c-lkcjk\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.583480 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.583447 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:03.962808 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:03.962783 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk"] Oct 30 20:39:03.964636 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:39:03.964609 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99b3c6d1_1ca9_4102_a550_2c0cb353f18c.slice/crio-68128c789f232ae10d559f273ea362bf600f0e1e2f6cd922f7f895ef55845fbc WatchSource:0}: Error finding container 68128c789f232ae10d559f273ea362bf600f0e1e2f6cd922f7f895ef55845fbc: Status 404 returned error can't find the container with id 68128c789f232ae10d559f273ea362bf600f0e1e2f6cd922f7f895ef55845fbc Oct 30 20:39:04.687643 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:04.687610 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" event={"ID":"99b3c6d1-1ca9-4102-a550-2c0cb353f18c","Type":"ContainerStarted","Data":"cc55acd2c3f0a5f7001fb183751c9ddba9459921c04bdb14bb906175700011d5"} Oct 30 20:39:04.688116 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:04.687652 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" event={"ID":"99b3c6d1-1ca9-4102-a550-2c0cb353f18c","Type":"ContainerStarted","Data":"68128c789f232ae10d559f273ea362bf600f0e1e2f6cd922f7f895ef55845fbc"} Oct 30 20:39:04.805930 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:04.805877 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podStartSLOduration=1.805863412 podStartE2EDuration="1.805863412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:04.80443749 +0000 UTC m=+707.730994405" watchObservedRunningTime="2026-10-30 20:39:04.805863412 +0000 UTC m=+707.732420337" Oct 30 20:39:05.584138 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:05.584106 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:11.254826 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:39:11.254787 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:39:13.583908 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:13.583878 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:13.601204 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:13.599568 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:13.749729 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:13.749700 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:39:18.691789 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.691758 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445"] Oct 30 20:39:18.696366 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.696349 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.759089 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.759059 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445"] Oct 30 20:39:18.801242 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.801211 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-tmp-dir\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.801242 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.801231 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ee3fad\"/\"premium-w6-ee3fad-kserve-self-signed-certs\"" Oct 30 20:39:18.801517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.801251 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-model-cache\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.801517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.801350 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d16f785a-baaa-466b-9965-51cf8a8a0bfe-tls-certs\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.801517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.801378 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-home\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.801517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.801462 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-dshm\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.801667 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.801521 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqshs\" (UniqueName: \"kubernetes.io/projected/d16f785a-baaa-466b-9965-51cf8a8a0bfe-kube-api-access-bqshs\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.902348 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.902309 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqshs\" (UniqueName: \"kubernetes.io/projected/d16f785a-baaa-466b-9965-51cf8a8a0bfe-kube-api-access-bqshs\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.902567 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.902364 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-tmp-dir\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.902567 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.902407 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-model-cache\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.902567 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.902446 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d16f785a-baaa-466b-9965-51cf8a8a0bfe-tls-certs\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.902567 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.902473 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-home\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.902567 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.902514 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-dshm\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.902845 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.902805 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-tmp-dir\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.902900 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.902858 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-model-cache\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.902900 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.902875 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-home\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.904847 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.904825 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-dshm\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.905191 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.905173 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d16f785a-baaa-466b-9965-51cf8a8a0bfe-tls-certs\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:18.961645 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:18.961576 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqshs\" (UniqueName: \"kubernetes.io/projected/d16f785a-baaa-466b-9965-51cf8a8a0bfe-kube-api-access-bqshs\") pod \"premium-w6-ee3fad-kserve-878b87998-l6445\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:19.007971 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:19.007943 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:19.386889 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:19.386866 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445"] Oct 30 20:39:19.389292 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:39:19.389259 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd16f785a_baaa_466b_9965_51cf8a8a0bfe.slice/crio-e23d293cf3d0d51d115b2072279445f3dfdf17cde41323132ec13d772255920e WatchSource:0}: Error finding container e23d293cf3d0d51d115b2072279445f3dfdf17cde41323132ec13d772255920e: Status 404 returned error can't find the container with id e23d293cf3d0d51d115b2072279445f3dfdf17cde41323132ec13d772255920e Oct 30 20:39:19.769840 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:19.769802 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" event={"ID":"d16f785a-baaa-466b-9965-51cf8a8a0bfe","Type":"ContainerStarted","Data":"99214e5c1a44b99a33692e72745e9537685c5e4d8ea90c51613b8e908ec6f8bb"} Oct 30 20:39:19.769840 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:19.769844 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" event={"ID":"d16f785a-baaa-466b-9965-51cf8a8a0bfe","Type":"ContainerStarted","Data":"e23d293cf3d0d51d115b2072279445f3dfdf17cde41323132ec13d772255920e"} Oct 30 20:39:19.885737 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:19.885686 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podStartSLOduration=1.885672348 podStartE2EDuration="1.885672348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:19.88492008 +0000 UTC m=+722.811477017" watchObservedRunningTime="2026-10-30 20:39:19.885672348 +0000 UTC m=+722.812229263" Oct 30 20:39:20.008757 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:20.008727 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:29.008576 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:29.008492 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:29.021060 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:29.021032 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:29.830788 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:29.830762 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:39:41.254908 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:39:41.254872 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:39:46.430429 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.430380 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7"] Oct 30 20:39:46.433699 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.433682 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.470594 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.470560 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-dshm\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.470761 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.470607 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9q4r\" (UniqueName: \"kubernetes.io/projected/68b1092f-af0f-477c-81c0-cc3083b7f564-kube-api-access-j9q4r\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.470761 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.470683 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-home\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.470761 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.470731 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-tmp-dir\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.470761 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.470750 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b1092f-af0f-477c-81c0-cc3083b7f564-tls-certs\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.470914 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.470774 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-model-cache\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.537910 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.537875 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f6ebf5\"/\"premium-w3-f6ebf5-kserve-self-signed-certs\"" Oct 30 20:39:46.539757 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.539732 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7"] Oct 30 20:39:46.571514 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.571479 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-dshm\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.571691 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.571536 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9q4r\" (UniqueName: \"kubernetes.io/projected/68b1092f-af0f-477c-81c0-cc3083b7f564-kube-api-access-j9q4r\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.571691 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.571582 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-home\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.571691 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.571658 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-tmp-dir\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.571863 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.571694 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b1092f-af0f-477c-81c0-cc3083b7f564-tls-certs\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.571863 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.571741 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-model-cache\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.572243 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.572218 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-model-cache\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.576858 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.576824 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-dshm\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.577658 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.577635 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-home\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.578991 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.578961 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-tmp-dir\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.582371 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.582327 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b1092f-af0f-477c-81c0-cc3083b7f564-tls-certs\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.643467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.643433 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9q4r\" (UniqueName: \"kubernetes.io/projected/68b1092f-af0f-477c-81c0-cc3083b7f564-kube-api-access-j9q4r\") pod \"premium-w3-f6ebf5-kserve-dfb898845-kvws7\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:46.745236 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:46.745199 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:47.140177 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:47.140156 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7"] Oct 30 20:39:47.142249 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:39:47.142218 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68b1092f_af0f_477c_81c0_cc3083b7f564.slice/crio-19bd8766b1a2951bec3e8f44988653b5ada7dae665eca6f9be00ce4365ea8241 WatchSource:0}: Error finding container 19bd8766b1a2951bec3e8f44988653b5ada7dae665eca6f9be00ce4365ea8241: Status 404 returned error can't find the container with id 19bd8766b1a2951bec3e8f44988653b5ada7dae665eca6f9be00ce4365ea8241 Oct 30 20:39:47.910864 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:47.910829 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" event={"ID":"68b1092f-af0f-477c-81c0-cc3083b7f564","Type":"ContainerStarted","Data":"c7a0c66c46121bdf79f94f1f7b6d82499d86ee878366b0f51b7f19c482aa6328"} Oct 30 20:39:47.911287 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:47.910875 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" event={"ID":"68b1092f-af0f-477c-81c0-cc3083b7f564","Type":"ContainerStarted","Data":"19bd8766b1a2951bec3e8f44988653b5ada7dae665eca6f9be00ce4365ea8241"} Oct 30 20:39:48.029675 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:48.029613 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podStartSLOduration=2.029594681 podStartE2EDuration="2.029594681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:48.028994085 +0000 UTC m=+750.955551002" watchObservedRunningTime="2026-10-30 20:39:48.029594681 +0000 UTC m=+750.956151597" Oct 30 20:39:48.746195 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:48.746168 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:56.746244 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:56.746208 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:56.759753 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:56.759724 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:39:56.967587 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:39:56.967559 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:40:11.254022 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:40:11.253985 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:40:41.254252 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:40:41.254212 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:40:53.367759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.367717 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq"] Oct 30 20:40:53.371558 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.371533 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.430147 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.430112 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq"] Oct 30 20:40:53.476836 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.476804 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-86d620\"/\"openshift-service-ca.crt\"" Oct 30 20:40:53.477013 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.476990 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-86d620\"/\"kube-root-ca.crt\"" Oct 30 20:40:53.477144 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.477065 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-86d620\"/\"premium-w1-86d620-kserve-self-signed-certs\"" Oct 30 20:40:53.477144 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.477065 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-86d620\"/\"default-dockercfg-sx62s\"" Oct 30 20:40:53.504942 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.504911 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcg49\" (UniqueName: \"kubernetes.io/projected/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-kube-api-access-dcg49\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.505088 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.504952 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-tmp-dir\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.505088 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.505041 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-model-cache\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.505088 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.505082 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-tls-certs\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.505191 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.505142 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-dshm\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.505191 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.505169 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-home\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.606242 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.606205 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-model-cache\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.606242 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.606244 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-tls-certs\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.606527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.606297 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-dshm\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.606527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.606334 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-home\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.606527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.606449 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcg49\" (UniqueName: \"kubernetes.io/projected/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-kube-api-access-dcg49\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.606527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.606491 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-tmp-dir\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.606795 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.606773 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-model-cache\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.606855 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.606839 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-home\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.606906 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.606867 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-tmp-dir\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.608780 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.608750 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-dshm\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.608979 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.608958 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-tls-certs\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.665482 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.665377 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcg49\" (UniqueName: \"kubernetes.io/projected/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-kube-api-access-dcg49\") pod \"premium-w1-86d620-kserve-bf694fd9f-ccrrq\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:53.706278 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:53.706242 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:40:54.086535 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:54.086510 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq"] Oct 30 20:40:54.087864 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:40:54.087838 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9614ceb3_8b56_4eb6_84cf_8904e13a1dda.slice/crio-d0b9adcce6f3376b9d0f8dcbbda9d3fcfb0b487110fd759beb5bf98977d019eb WatchSource:0}: Error finding container d0b9adcce6f3376b9d0f8dcbbda9d3fcfb0b487110fd759beb5bf98977d019eb: Status 404 returned error can't find the container with id d0b9adcce6f3376b9d0f8dcbbda9d3fcfb0b487110fd759beb5bf98977d019eb Oct 30 20:40:54.254951 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:54.254915 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" event={"ID":"9614ceb3-8b56-4eb6-84cf-8904e13a1dda","Type":"ContainerStarted","Data":"acdb8fad8dac50a4b7fefff060442ff12cdbcbf9d2370d08b342716af9eadce1"} Oct 30 20:40:54.255151 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:54.254958 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" event={"ID":"9614ceb3-8b56-4eb6-84cf-8904e13a1dda","Type":"ContainerStarted","Data":"d0b9adcce6f3376b9d0f8dcbbda9d3fcfb0b487110fd759beb5bf98977d019eb"} Oct 30 20:40:54.372800 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:54.372696 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podStartSLOduration=1.372682027 podStartE2EDuration="1.372682027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:54.370681256 +0000 UTC m=+817.297238187" watchObservedRunningTime="2026-10-30 20:40:54.372682027 +0000 UTC m=+817.299238939" Oct 30 20:40:54.707314 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:40:54.707232 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:41:03.707026 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:03.706944 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:41:03.719492 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:03.719467 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:41:04.317500 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:04.317473 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:41:11.255361 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:41:11.255252 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:41:41.253827 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:41:41.253791 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:41:58.733467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.733422 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b"] Oct 30 20:41:58.737191 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.737169 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.793373 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.793344 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-386743-openshift-default-dockercfg-zfbp9\"" Oct 30 20:41:58.838845 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.838810 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjn7l\" (UniqueName: \"kubernetes.io/projected/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-kube-api-access-cjn7l\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.838998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.838876 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-credential-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.838998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.838923 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istiod-ca-cert\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.839068 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.839000 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-podinfo\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.839068 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.839032 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-workload-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.839132 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.839076 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-data\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.839132 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.839121 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-envoy\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.839191 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.839152 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-workload-certs\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.839191 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.839176 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-token\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.844609 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.844589 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b"] Oct 30 20:41:58.940279 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.940243 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjn7l\" (UniqueName: \"kubernetes.io/projected/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-kube-api-access-cjn7l\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.940516 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.940332 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-credential-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.940516 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.940354 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istiod-ca-cert\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.940516 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.940377 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-podinfo\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.940516 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.940433 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-workload-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.940516 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.940457 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-data\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.940803 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.940518 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-envoy\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.940803 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.940566 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-workload-certs\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.940803 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.940602 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-token\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.940968 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.940852 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-workload-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.940968 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.940856 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-credential-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.940968 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.940933 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-data\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.940968 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.940943 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-workload-certs\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.941254 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.941237 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istiod-ca-cert\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.942986 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.942957 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-envoy\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.943105 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.942987 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-podinfo\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.998996 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.998940 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-token\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:58.998996 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:58.998988 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjn7l\" (UniqueName: \"kubernetes.io/projected/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-kube-api-access-cjn7l\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:59.049831 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:59.049806 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:41:59.478849 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:59.478819 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b"] Oct 30 20:41:59.480289 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:41:59.480263 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80aa1317_6c2c_48b4_a5f5_bbfa7928f08a.slice/crio-26f2941eaf04c3a2800d4e1d15196db626c47e6d24e9ac4c20187f7741934885 WatchSource:0}: Error finding container 26f2941eaf04c3a2800d4e1d15196db626c47e6d24e9ac4c20187f7741934885: Status 404 returned error can't find the container with id 26f2941eaf04c3a2800d4e1d15196db626c47e6d24e9ac4c20187f7741934885 Oct 30 20:41:59.482768 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:59.482735 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:41:59.482844 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:59.482801 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:41:59.482892 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:59.482847 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:41:59.599452 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:59.599419 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" event={"ID":"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a","Type":"ContainerStarted","Data":"9cda15cfe7c97261babd672c204d36facc1f1b977bc65661a298e2eb33529ca8"} Oct 30 20:41:59.599615 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:59.599475 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" event={"ID":"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a","Type":"ContainerStarted","Data":"26f2941eaf04c3a2800d4e1d15196db626c47e6d24e9ac4c20187f7741934885"} Oct 30 20:41:59.719300 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:41:59.719237 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" podStartSLOduration=1.719220677 podStartE2EDuration="1.719220677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:59.71842067 +0000 UTC m=+882.644977604" watchObservedRunningTime="2026-10-30 20:41:59.719220677 +0000 UTC m=+882.645777592" Oct 30 20:42:00.050836 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:00.050801 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:42:00.052099 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:00.052076 5755 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:42:00.052235 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:00.052128 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" podUID="80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:42:01.050851 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:01.050815 5755 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:42:01.051238 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:01.050879 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" podUID="80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:42:02.050569 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:02.050533 5755 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:42:02.050747 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:02.050585 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" podUID="80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:42:03.050736 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:03.050702 5755 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:42:03.051119 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:03.050754 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" podUID="80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:42:04.050842 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:04.050808 5755 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:42:04.051249 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:04.050869 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" podUID="80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:42:05.050785 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:05.050742 5755 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:42:05.050980 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:05.050808 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" podUID="80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:42:06.054510 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:06.054483 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:42:06.054894 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:06.054876 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:42:06.055572 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:06.055552 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:42:10.332718 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:10.332683 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj"] Oct 30 20:42:10.336491 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:10.336466 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" Oct 30 20:42:10.393292 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:10.393261 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj"] Oct 30 20:42:10.440335 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:10.440307 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:42:10.440335 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:10.440322 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:42:10.480790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:10.480760 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5396a0db-8f86-4fee-b99d-f629526be15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a5961ce-pjvfj\" (UID: \"5396a0db-8f86-4fee-b99d-f629526be15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" Oct 30 20:42:10.480936 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:10.480799 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5396a0db-8f86-4fee-b99d-f629526be15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a5961ce-pjvfj\" (UID: \"5396a0db-8f86-4fee-b99d-f629526be15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" Oct 30 20:42:10.480936 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:10.480906 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xlb4\" (UniqueName: \"kubernetes.io/projected/5396a0db-8f86-4fee-b99d-f629526be15d-kube-api-access-6xlb4\") pod \"maas-api-revoke-sub-3a5961ce-pjvfj\" (UID: \"5396a0db-8f86-4fee-b99d-f629526be15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" Oct 30 20:42:10.581684 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:10.581652 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5396a0db-8f86-4fee-b99d-f629526be15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a5961ce-pjvfj\" (UID: \"5396a0db-8f86-4fee-b99d-f629526be15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" Oct 30 20:42:10.581876 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:10.581701 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5396a0db-8f86-4fee-b99d-f629526be15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a5961ce-pjvfj\" (UID: \"5396a0db-8f86-4fee-b99d-f629526be15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" Oct 30 20:42:10.581876 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:10.581765 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xlb4\" (UniqueName: \"kubernetes.io/projected/5396a0db-8f86-4fee-b99d-f629526be15d-kube-api-access-6xlb4\") pod \"maas-api-revoke-sub-3a5961ce-pjvfj\" (UID: \"5396a0db-8f86-4fee-b99d-f629526be15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" Oct 30 20:42:10.582636 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:10.582584 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5396a0db-8f86-4fee-b99d-f629526be15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a5961ce-pjvfj\" (UID: \"5396a0db-8f86-4fee-b99d-f629526be15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" Oct 30 20:42:10.584171 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:10.584122 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5396a0db-8f86-4fee-b99d-f629526be15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a5961ce-pjvfj\" (UID: \"5396a0db-8f86-4fee-b99d-f629526be15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" Oct 30 20:42:10.640988 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:10.640956 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xlb4\" (UniqueName: \"kubernetes.io/projected/5396a0db-8f86-4fee-b99d-f629526be15d-kube-api-access-6xlb4\") pod \"maas-api-revoke-sub-3a5961ce-pjvfj\" (UID: \"5396a0db-8f86-4fee-b99d-f629526be15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" Oct 30 20:42:10.648899 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:10.648882 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" Oct 30 20:42:11.023559 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:11.023537 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj"] Oct 30 20:42:11.025233 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:42:11.025205 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5396a0db_8f86_4fee_b99d_f629526be15d.slice/crio-8128960007cc24ca2ef4c80fa501c93bdd3e5ae590200e95d052c9332d656352 WatchSource:0}: Error finding container 8128960007cc24ca2ef4c80fa501c93bdd3e5ae590200e95d052c9332d656352: Status 404 returned error can't find the container with id 8128960007cc24ca2ef4c80fa501c93bdd3e5ae590200e95d052c9332d656352 Oct 30 20:42:11.255111 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:42:11.255076 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:42:11.661656 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:11.661624 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" event={"ID":"5396a0db-8f86-4fee-b99d-f629526be15d","Type":"ContainerStarted","Data":"8128960007cc24ca2ef4c80fa501c93bdd3e5ae590200e95d052c9332d656352"} Oct 30 20:42:15.692552 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:15.692512 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" event={"ID":"5396a0db-8f86-4fee-b99d-f629526be15d","Type":"ContainerStarted","Data":"b0a9631a3017254568f68669cf360f135322163feab8936aeb862b5e7f5bd420"} Oct 30 20:42:15.809949 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:15.809897 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" podStartSLOduration=1.575244807 podStartE2EDuration="5.809883041s" totalImagesPullingTime="4.234638234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:15.808361867 +0000 UTC m=+898.734918792" watchObservedRunningTime="2026-10-30 20:42:15.809883041 +0000 UTC m=+898.736439956" Oct 30 20:42:16.701884 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:16.701846 5755 generic.go:356] "Generic (PLEG): container finished" podID="5396a0db-8f86-4fee-b99d-f629526be15d" containerID="b0a9631a3017254568f68669cf360f135322163feab8936aeb862b5e7f5bd420" exitCode=0 Oct 30 20:42:16.702307 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:16.701945 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" event={"ID":"5396a0db-8f86-4fee-b99d-f629526be15d","Type":"ContainerDied","Data":"b0a9631a3017254568f68669cf360f135322163feab8936aeb862b5e7f5bd420"} Oct 30 20:42:17.845245 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:17.845208 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" Oct 30 20:42:17.956536 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:17.956508 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5396a0db-8f86-4fee-b99d-f629526be15d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5396a0db-8f86-4fee-b99d-f629526be15d-maas-api-xks-ca\") pod \"5396a0db-8f86-4fee-b99d-f629526be15d\" (UID: \"5396a0db-8f86-4fee-b99d-f629526be15d\") " Oct 30 20:42:17.956692 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:17.956587 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5396a0db-8f86-4fee-b99d-f629526be15d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5396a0db-8f86-4fee-b99d-f629526be15d-maas-api-openshift-ca\") pod \"5396a0db-8f86-4fee-b99d-f629526be15d\" (UID: \"5396a0db-8f86-4fee-b99d-f629526be15d\") " Oct 30 20:42:17.956747 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:17.956704 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5396a0db-8f86-4fee-b99d-f629526be15d-kube-api-access-6xlb4\" (UniqueName: \"kubernetes.io/projected/5396a0db-8f86-4fee-b99d-f629526be15d-kube-api-access-6xlb4\") pod \"5396a0db-8f86-4fee-b99d-f629526be15d\" (UID: \"5396a0db-8f86-4fee-b99d-f629526be15d\") " Oct 30 20:42:17.956974 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:17.956944 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5396a0db-8f86-4fee-b99d-f629526be15d-maas-api-openshift-ca" pod "5396a0db-8f86-4fee-b99d-f629526be15d" (UID: "5396a0db-8f86-4fee-b99d-f629526be15d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:42:17.957100 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:17.957037 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5396a0db-8f86-4fee-b99d-f629526be15d-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:42:17.958828 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:17.958774 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5396a0db-8f86-4fee-b99d-f629526be15d-maas-api-xks-ca" pod "5396a0db-8f86-4fee-b99d-f629526be15d" (UID: "5396a0db-8f86-4fee-b99d-f629526be15d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:42:17.958828 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:17.958797 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5396a0db-8f86-4fee-b99d-f629526be15d-kube-api-access-6xlb4" pod "5396a0db-8f86-4fee-b99d-f629526be15d" (UID: "5396a0db-8f86-4fee-b99d-f629526be15d"). InnerVolumeSpecName "kube-api-access-6xlb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:42:18.057899 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:18.057864 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xlb4\" (UniqueName: \"kubernetes.io/projected/5396a0db-8f86-4fee-b99d-f629526be15d-kube-api-access-6xlb4\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:42:18.057899 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:18.057895 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5396a0db-8f86-4fee-b99d-f629526be15d-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:42:18.713801 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:18.713767 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" Oct 30 20:42:18.713801 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:18.713789 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj" event={"ID":"5396a0db-8f86-4fee-b99d-f629526be15d","Type":"ContainerDied","Data":"8128960007cc24ca2ef4c80fa501c93bdd3e5ae590200e95d052c9332d656352"} Oct 30 20:42:18.714090 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:18.713842 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8128960007cc24ca2ef4c80fa501c93bdd3e5ae590200e95d052c9332d656352" Oct 30 20:42:27.347693 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.347610 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h"] Oct 30 20:42:27.348089 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.348023 5755 cpu_manager.go:414] "containerMap: removing container" podUID="5396a0db-8f86-4fee-b99d-f629526be15d" containerName="revoke-keys" Oct 30 20:42:27.348089 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.348035 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5396a0db-8f86-4fee-b99d-f629526be15d" containerName="revoke-keys" Oct 30 20:42:27.348196 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.348118 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="5396a0db-8f86-4fee-b99d-f629526be15d" containerName="revoke-keys" Oct 30 20:42:27.351289 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.351267 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" Oct 30 20:42:27.444332 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.444301 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3de445e8-8a06-4000-bfb9-be20213a0b12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a169dad7-pkz5h\" (UID: \"3de445e8-8a06-4000-bfb9-be20213a0b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" Oct 30 20:42:27.444509 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.444351 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d6q7\" (UniqueName: \"kubernetes.io/projected/3de445e8-8a06-4000-bfb9-be20213a0b12-kube-api-access-9d6q7\") pod \"maas-api-revoke-sub-a169dad7-pkz5h\" (UID: \"3de445e8-8a06-4000-bfb9-be20213a0b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" Oct 30 20:42:27.444568 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.444503 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3de445e8-8a06-4000-bfb9-be20213a0b12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a169dad7-pkz5h\" (UID: \"3de445e8-8a06-4000-bfb9-be20213a0b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" Oct 30 20:42:27.454030 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.453998 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:42:27.454030 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.454021 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:42:27.456287 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.456269 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h"] Oct 30 20:42:27.546040 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.546010 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3de445e8-8a06-4000-bfb9-be20213a0b12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a169dad7-pkz5h\" (UID: \"3de445e8-8a06-4000-bfb9-be20213a0b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" Oct 30 20:42:27.546188 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.546046 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d6q7\" (UniqueName: \"kubernetes.io/projected/3de445e8-8a06-4000-bfb9-be20213a0b12-kube-api-access-9d6q7\") pod \"maas-api-revoke-sub-a169dad7-pkz5h\" (UID: \"3de445e8-8a06-4000-bfb9-be20213a0b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" Oct 30 20:42:27.546188 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.546081 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3de445e8-8a06-4000-bfb9-be20213a0b12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a169dad7-pkz5h\" (UID: \"3de445e8-8a06-4000-bfb9-be20213a0b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" Oct 30 20:42:27.546875 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.546780 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3de445e8-8a06-4000-bfb9-be20213a0b12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a169dad7-pkz5h\" (UID: \"3de445e8-8a06-4000-bfb9-be20213a0b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" Oct 30 20:42:27.548522 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.548501 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3de445e8-8a06-4000-bfb9-be20213a0b12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a169dad7-pkz5h\" (UID: \"3de445e8-8a06-4000-bfb9-be20213a0b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" Oct 30 20:42:27.605558 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.605486 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d6q7\" (UniqueName: \"kubernetes.io/projected/3de445e8-8a06-4000-bfb9-be20213a0b12-kube-api-access-9d6q7\") pod \"maas-api-revoke-sub-a169dad7-pkz5h\" (UID: \"3de445e8-8a06-4000-bfb9-be20213a0b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" Oct 30 20:42:27.664174 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:27.664142 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" Oct 30 20:42:28.046286 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:28.046258 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h"] Oct 30 20:42:28.047308 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:42:28.047283 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de445e8_8a06_4000_bfb9_be20213a0b12.slice/crio-4599a65313301a17d82d82e0f4f2d2348bddcf891ed802113a4282030c621711 WatchSource:0}: Error finding container 4599a65313301a17d82d82e0f4f2d2348bddcf891ed802113a4282030c621711: Status 404 returned error can't find the container with id 4599a65313301a17d82d82e0f4f2d2348bddcf891ed802113a4282030c621711 Oct 30 20:42:28.766100 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:28.766069 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" event={"ID":"3de445e8-8a06-4000-bfb9-be20213a0b12","Type":"ContainerStarted","Data":"20cfb2f6ce1752dc13033700df7c96e3e173d0efc70af11d9175dcdbae3d232e"} Oct 30 20:42:28.766529 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:28.766118 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" event={"ID":"3de445e8-8a06-4000-bfb9-be20213a0b12","Type":"ContainerStarted","Data":"4599a65313301a17d82d82e0f4f2d2348bddcf891ed802113a4282030c621711"} Oct 30 20:42:28.892123 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:28.892040 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" podStartSLOduration=1.892024174 podStartE2EDuration="1.892024174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:28.891076977 +0000 UTC m=+911.817633894" watchObservedRunningTime="2026-10-30 20:42:28.892024174 +0000 UTC m=+911.818581089" Oct 30 20:42:29.773108 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:29.772992 5755 generic.go:356] "Generic (PLEG): container finished" podID="3de445e8-8a06-4000-bfb9-be20213a0b12" containerID="20cfb2f6ce1752dc13033700df7c96e3e173d0efc70af11d9175dcdbae3d232e" exitCode=0 Oct 30 20:42:29.773108 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:29.773034 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" event={"ID":"3de445e8-8a06-4000-bfb9-be20213a0b12","Type":"ContainerDied","Data":"20cfb2f6ce1752dc13033700df7c96e3e173d0efc70af11d9175dcdbae3d232e"} Oct 30 20:42:30.180441 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.180360 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7"] Oct 30 20:42:30.183796 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.183781 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7" Oct 30 20:42:30.251458 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.251429 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7"] Oct 30 20:42:30.275066 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.275031 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgnk2\" (UniqueName: \"kubernetes.io/projected/8cf3a0d3-0027-4206-8b22-2f91f716329c-kube-api-access-xgnk2\") pod \"maas-api-revoke-sub-b9483bf5-fgmx7\" (UID: \"8cf3a0d3-0027-4206-8b22-2f91f716329c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7" Oct 30 20:42:30.275231 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.275081 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf3a0d3-0027-4206-8b22-2f91f716329c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9483bf5-fgmx7\" (UID: \"8cf3a0d3-0027-4206-8b22-2f91f716329c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7" Oct 30 20:42:30.275231 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.275118 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf3a0d3-0027-4206-8b22-2f91f716329c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9483bf5-fgmx7\" (UID: \"8cf3a0d3-0027-4206-8b22-2f91f716329c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7" Oct 30 20:42:30.376082 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.376049 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnk2\" (UniqueName: \"kubernetes.io/projected/8cf3a0d3-0027-4206-8b22-2f91f716329c-kube-api-access-xgnk2\") pod \"maas-api-revoke-sub-b9483bf5-fgmx7\" (UID: \"8cf3a0d3-0027-4206-8b22-2f91f716329c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7" Oct 30 20:42:30.376274 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.376096 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf3a0d3-0027-4206-8b22-2f91f716329c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9483bf5-fgmx7\" (UID: \"8cf3a0d3-0027-4206-8b22-2f91f716329c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7" Oct 30 20:42:30.376274 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.376127 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf3a0d3-0027-4206-8b22-2f91f716329c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9483bf5-fgmx7\" (UID: \"8cf3a0d3-0027-4206-8b22-2f91f716329c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7" Oct 30 20:42:30.376964 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.376874 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf3a0d3-0027-4206-8b22-2f91f716329c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9483bf5-fgmx7\" (UID: \"8cf3a0d3-0027-4206-8b22-2f91f716329c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7" Oct 30 20:42:30.378559 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.378535 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf3a0d3-0027-4206-8b22-2f91f716329c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9483bf5-fgmx7\" (UID: \"8cf3a0d3-0027-4206-8b22-2f91f716329c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7" Oct 30 20:42:30.436445 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.436343 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgnk2\" (UniqueName: \"kubernetes.io/projected/8cf3a0d3-0027-4206-8b22-2f91f716329c-kube-api-access-xgnk2\") pod \"maas-api-revoke-sub-b9483bf5-fgmx7\" (UID: \"8cf3a0d3-0027-4206-8b22-2f91f716329c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7" Oct 30 20:42:30.496550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.496521 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7" Oct 30 20:42:30.875989 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.875955 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7"] Oct 30 20:42:30.879705 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:42:30.879681 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf3a0d3_0027_4206_8b22_2f91f716329c.slice/crio-10ff90b1e05d42346c1f211183d2451a3e7402422636d006b7513c5a07d9f83e WatchSource:0}: Error finding container 10ff90b1e05d42346c1f211183d2451a3e7402422636d006b7513c5a07d9f83e: Status 404 returned error can't find the container with id 10ff90b1e05d42346c1f211183d2451a3e7402422636d006b7513c5a07d9f83e Oct 30 20:42:30.923203 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.923178 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" Oct 30 20:42:30.983404 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.983366 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3de445e8-8a06-4000-bfb9-be20213a0b12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3de445e8-8a06-4000-bfb9-be20213a0b12-maas-api-xks-ca\") pod \"3de445e8-8a06-4000-bfb9-be20213a0b12\" (UID: \"3de445e8-8a06-4000-bfb9-be20213a0b12\") " Oct 30 20:42:30.983554 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.983466 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de445e8-8a06-4000-bfb9-be20213a0b12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3de445e8-8a06-4000-bfb9-be20213a0b12-maas-api-openshift-ca\") pod \"3de445e8-8a06-4000-bfb9-be20213a0b12\" (UID: \"3de445e8-8a06-4000-bfb9-be20213a0b12\") " Oct 30 20:42:30.983554 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.983526 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de445e8-8a06-4000-bfb9-be20213a0b12-kube-api-access-9d6q7\" (UniqueName: \"kubernetes.io/projected/3de445e8-8a06-4000-bfb9-be20213a0b12-kube-api-access-9d6q7\") pod \"3de445e8-8a06-4000-bfb9-be20213a0b12\" (UID: \"3de445e8-8a06-4000-bfb9-be20213a0b12\") " Oct 30 20:42:30.983880 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.983853 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de445e8-8a06-4000-bfb9-be20213a0b12-maas-api-openshift-ca" pod "3de445e8-8a06-4000-bfb9-be20213a0b12" (UID: "3de445e8-8a06-4000-bfb9-be20213a0b12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:42:30.985769 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.985747 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de445e8-8a06-4000-bfb9-be20213a0b12-maas-api-xks-ca" pod "3de445e8-8a06-4000-bfb9-be20213a0b12" (UID: "3de445e8-8a06-4000-bfb9-be20213a0b12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:42:30.985943 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:30.985927 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de445e8-8a06-4000-bfb9-be20213a0b12-kube-api-access-9d6q7" pod "3de445e8-8a06-4000-bfb9-be20213a0b12" (UID: "3de445e8-8a06-4000-bfb9-be20213a0b12"). InnerVolumeSpecName "kube-api-access-9d6q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:42:31.085183 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:31.085116 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3de445e8-8a06-4000-bfb9-be20213a0b12-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:42:31.085183 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:31.085145 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3de445e8-8a06-4000-bfb9-be20213a0b12-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:42:31.085183 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:31.085159 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d6q7\" (UniqueName: \"kubernetes.io/projected/3de445e8-8a06-4000-bfb9-be20213a0b12-kube-api-access-9d6q7\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:42:31.785759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:31.785725 5755 generic.go:356] "Generic (PLEG): container finished" podID="8cf3a0d3-0027-4206-8b22-2f91f716329c" containerID="a62fa543769404233279205da405f8c7fd0b7e6a49b143113f05e2559b94f2ea" exitCode=0 Oct 30 20:42:31.785946 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:31.785792 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7" event={"ID":"8cf3a0d3-0027-4206-8b22-2f91f716329c","Type":"ContainerDied","Data":"a62fa543769404233279205da405f8c7fd0b7e6a49b143113f05e2559b94f2ea"} Oct 30 20:42:31.785946 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:31.785855 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7" event={"ID":"8cf3a0d3-0027-4206-8b22-2f91f716329c","Type":"ContainerStarted","Data":"10ff90b1e05d42346c1f211183d2451a3e7402422636d006b7513c5a07d9f83e"} Oct 30 20:42:31.787457 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:31.787431 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" event={"ID":"3de445e8-8a06-4000-bfb9-be20213a0b12","Type":"ContainerDied","Data":"4599a65313301a17d82d82e0f4f2d2348bddcf891ed802113a4282030c621711"} Oct 30 20:42:31.787569 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:31.787462 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h" Oct 30 20:42:31.787569 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:31.787478 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4599a65313301a17d82d82e0f4f2d2348bddcf891ed802113a4282030c621711" Oct 30 20:42:32.939188 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:32.939162 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7" Oct 30 20:42:33.004404 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:33.004360 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cf3a0d3-0027-4206-8b22-2f91f716329c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf3a0d3-0027-4206-8b22-2f91f716329c-maas-api-xks-ca\") pod \"8cf3a0d3-0027-4206-8b22-2f91f716329c\" (UID: \"8cf3a0d3-0027-4206-8b22-2f91f716329c\") " Oct 30 20:42:33.004541 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:33.004411 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf3a0d3-0027-4206-8b22-2f91f716329c-kube-api-access-xgnk2\" (UniqueName: \"kubernetes.io/projected/8cf3a0d3-0027-4206-8b22-2f91f716329c-kube-api-access-xgnk2\") pod \"8cf3a0d3-0027-4206-8b22-2f91f716329c\" (UID: \"8cf3a0d3-0027-4206-8b22-2f91f716329c\") " Oct 30 20:42:33.004541 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:33.004458 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf3a0d3-0027-4206-8b22-2f91f716329c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf3a0d3-0027-4206-8b22-2f91f716329c-maas-api-openshift-ca\") pod \"8cf3a0d3-0027-4206-8b22-2f91f716329c\" (UID: \"8cf3a0d3-0027-4206-8b22-2f91f716329c\") " Oct 30 20:42:33.004875 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:33.004851 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf3a0d3-0027-4206-8b22-2f91f716329c-maas-api-openshift-ca" pod "8cf3a0d3-0027-4206-8b22-2f91f716329c" (UID: "8cf3a0d3-0027-4206-8b22-2f91f716329c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:42:33.006470 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:33.006446 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf3a0d3-0027-4206-8b22-2f91f716329c-maas-api-xks-ca" pod "8cf3a0d3-0027-4206-8b22-2f91f716329c" (UID: "8cf3a0d3-0027-4206-8b22-2f91f716329c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:42:33.006586 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:33.006567 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf3a0d3-0027-4206-8b22-2f91f716329c-kube-api-access-xgnk2" pod "8cf3a0d3-0027-4206-8b22-2f91f716329c" (UID: "8cf3a0d3-0027-4206-8b22-2f91f716329c"). InnerVolumeSpecName "kube-api-access-xgnk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:42:33.106226 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:33.106159 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf3a0d3-0027-4206-8b22-2f91f716329c-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:42:33.106226 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:33.106185 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgnk2\" (UniqueName: \"kubernetes.io/projected/8cf3a0d3-0027-4206-8b22-2f91f716329c-kube-api-access-xgnk2\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:42:33.106226 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:33.106195 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf3a0d3-0027-4206-8b22-2f91f716329c-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:42:33.799701 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:33.799674 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7" Oct 30 20:42:33.799884 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:33.799714 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7" event={"ID":"8cf3a0d3-0027-4206-8b22-2f91f716329c","Type":"ContainerDied","Data":"10ff90b1e05d42346c1f211183d2451a3e7402422636d006b7513c5a07d9f83e"} Oct 30 20:42:33.799884 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:33.799762 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10ff90b1e05d42346c1f211183d2451a3e7402422636d006b7513c5a07d9f83e" Oct 30 20:42:36.003471 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.003441 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64"] Oct 30 20:42:36.003882 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.003867 5755 cpu_manager.go:414] "containerMap: removing container" podUID="8cf3a0d3-0027-4206-8b22-2f91f716329c" containerName="revoke-keys" Oct 30 20:42:36.003931 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.003886 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf3a0d3-0027-4206-8b22-2f91f716329c" containerName="revoke-keys" Oct 30 20:42:36.003931 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.003905 5755 cpu_manager.go:414] "containerMap: removing container" podUID="3de445e8-8a06-4000-bfb9-be20213a0b12" containerName="revoke-keys" Oct 30 20:42:36.003931 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.003910 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de445e8-8a06-4000-bfb9-be20213a0b12" containerName="revoke-keys" Oct 30 20:42:36.004026 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.003984 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf3a0d3-0027-4206-8b22-2f91f716329c" containerName="revoke-keys" Oct 30 20:42:36.004026 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.003994 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de445e8-8a06-4000-bfb9-be20213a0b12" containerName="revoke-keys" Oct 30 20:42:36.007265 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.007251 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.077171 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.077132 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64"] Oct 30 20:42:36.113015 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.112987 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b32941\"/\"openshift-service-ca.crt\"" Oct 30 20:42:36.113015 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.112995 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b32941\"/\"kube-root-ca.crt\"" Oct 30 20:42:36.113187 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.113137 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b32941\"/\"default-dockercfg-gqqws\"" Oct 30 20:42:36.113374 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.113359 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b32941\"/\"distinct-w0-b32941-kserve-self-signed-certs\"" Oct 30 20:42:36.136468 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.136440 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-home\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.136650 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.136520 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-dshm\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.136650 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.136541 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbzdx\" (UniqueName: \"kubernetes.io/projected/b2b57028-ba25-496d-834e-5433b95cc2a4-kube-api-access-gbzdx\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.136650 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.136582 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-model-cache\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.136650 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.136612 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b57028-ba25-496d-834e-5433b95cc2a4-tls-certs\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.136783 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.136680 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-tmp-dir\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.237413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.237349 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-model-cache\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.237601 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.237417 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b57028-ba25-496d-834e-5433b95cc2a4-tls-certs\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.237601 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.237465 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-tmp-dir\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.237601 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.237590 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-home\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.237771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.237631 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-dshm\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.237771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.237658 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbzdx\" (UniqueName: \"kubernetes.io/projected/b2b57028-ba25-496d-834e-5433b95cc2a4-kube-api-access-gbzdx\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.237771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.237730 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-model-cache\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.237914 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.237866 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-tmp-dir\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.237914 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.237892 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-home\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.239805 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.239784 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-dshm\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.240191 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.240168 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b57028-ba25-496d-834e-5433b95cc2a4-tls-certs\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.296659 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.296589 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbzdx\" (UniqueName: \"kubernetes.io/projected/b2b57028-ba25-496d-834e-5433b95cc2a4-kube-api-access-gbzdx\") pod \"distinct-w0-b32941-kserve-667ffcb56d-5dn64\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.319550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.319521 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:36.702842 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.702819 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64"] Oct 30 20:42:36.704513 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:42:36.704485 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2b57028_ba25_496d_834e_5433b95cc2a4.slice/crio-5d3db2d24539b5d1a510588a49f9aca17cef7e5479c9f817f1440cf718647bbd WatchSource:0}: Error finding container 5d3db2d24539b5d1a510588a49f9aca17cef7e5479c9f817f1440cf718647bbd: Status 404 returned error can't find the container with id 5d3db2d24539b5d1a510588a49f9aca17cef7e5479c9f817f1440cf718647bbd Oct 30 20:42:36.821560 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.821512 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" event={"ID":"b2b57028-ba25-496d-834e-5433b95cc2a4","Type":"ContainerStarted","Data":"48d4df7a0c35b7ede7363d91a660bbe248703e5ccc553ae7a3c39ea0361c9749"} Oct 30 20:42:36.821560 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.821561 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" event={"ID":"b2b57028-ba25-496d-834e-5433b95cc2a4","Type":"ContainerStarted","Data":"5d3db2d24539b5d1a510588a49f9aca17cef7e5479c9f817f1440cf718647bbd"} Oct 30 20:42:36.939018 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:36.938964 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podStartSLOduration=1.9389489530000001 podStartE2EDuration="1.938948953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:36.9376521 +0000 UTC m=+919.864209033" watchObservedRunningTime="2026-10-30 20:42:36.938948953 +0000 UTC m=+919.865505915" Oct 30 20:42:37.319968 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:37.319936 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:38.266165 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:38.266130 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d"] Oct 30 20:42:38.269782 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:38.269766 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d" Oct 30 20:42:38.328346 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:38.328314 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d"] Oct 30 20:42:38.359614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:38.359586 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m67x5\" (UniqueName: \"kubernetes.io/projected/cdba020f-8e2d-40d5-9f78-17970b545e80-kube-api-access-m67x5\") pod \"maas-api-revoke-sub-882a72e0-z6q4d\" (UID: \"cdba020f-8e2d-40d5-9f78-17970b545e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d" Oct 30 20:42:38.359777 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:38.359622 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdba020f-8e2d-40d5-9f78-17970b545e80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-882a72e0-z6q4d\" (UID: \"cdba020f-8e2d-40d5-9f78-17970b545e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d" Oct 30 20:42:38.359777 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:38.359765 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdba020f-8e2d-40d5-9f78-17970b545e80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-882a72e0-z6q4d\" (UID: \"cdba020f-8e2d-40d5-9f78-17970b545e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d" Oct 30 20:42:38.374376 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:38.374354 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:42:38.374493 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:38.374463 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:42:38.461214 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:38.461186 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdba020f-8e2d-40d5-9f78-17970b545e80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-882a72e0-z6q4d\" (UID: \"cdba020f-8e2d-40d5-9f78-17970b545e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d" Oct 30 20:42:38.461374 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:38.461281 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m67x5\" (UniqueName: \"kubernetes.io/projected/cdba020f-8e2d-40d5-9f78-17970b545e80-kube-api-access-m67x5\") pod \"maas-api-revoke-sub-882a72e0-z6q4d\" (UID: \"cdba020f-8e2d-40d5-9f78-17970b545e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d" Oct 30 20:42:38.461374 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:38.461302 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdba020f-8e2d-40d5-9f78-17970b545e80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-882a72e0-z6q4d\" (UID: \"cdba020f-8e2d-40d5-9f78-17970b545e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d" Oct 30 20:42:38.461899 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:38.461875 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdba020f-8e2d-40d5-9f78-17970b545e80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-882a72e0-z6q4d\" (UID: \"cdba020f-8e2d-40d5-9f78-17970b545e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d" Oct 30 20:42:38.463675 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:38.463645 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdba020f-8e2d-40d5-9f78-17970b545e80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-882a72e0-z6q4d\" (UID: \"cdba020f-8e2d-40d5-9f78-17970b545e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d" Oct 30 20:42:38.520699 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:38.520632 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m67x5\" (UniqueName: \"kubernetes.io/projected/cdba020f-8e2d-40d5-9f78-17970b545e80-kube-api-access-m67x5\") pod \"maas-api-revoke-sub-882a72e0-z6q4d\" (UID: \"cdba020f-8e2d-40d5-9f78-17970b545e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d" Oct 30 20:42:38.583183 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:38.583154 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d" Oct 30 20:42:38.960921 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:38.960898 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d"] Oct 30 20:42:38.962791 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:42:38.962759 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdba020f_8e2d_40d5_9f78_17970b545e80.slice/crio-92b0362febfe6bb286ed9a9647aaeecc4aa8f26f3ee12d41ec3366c6296eae22 WatchSource:0}: Error finding container 92b0362febfe6bb286ed9a9647aaeecc4aa8f26f3ee12d41ec3366c6296eae22: Status 404 returned error can't find the container with id 92b0362febfe6bb286ed9a9647aaeecc4aa8f26f3ee12d41ec3366c6296eae22 Oct 30 20:42:39.843188 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:39.843161 5755 generic.go:356] "Generic (PLEG): container finished" podID="cdba020f-8e2d-40d5-9f78-17970b545e80" containerID="3e25f9820ad45a897100bb3d3db3ca7afcc17cc3be3dc73dc7cf3090434d4f5a" exitCode=0 Oct 30 20:42:39.843538 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:39.843247 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d" event={"ID":"cdba020f-8e2d-40d5-9f78-17970b545e80","Type":"ContainerDied","Data":"3e25f9820ad45a897100bb3d3db3ca7afcc17cc3be3dc73dc7cf3090434d4f5a"} Oct 30 20:42:39.843538 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:39.843302 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d" event={"ID":"cdba020f-8e2d-40d5-9f78-17970b545e80","Type":"ContainerStarted","Data":"92b0362febfe6bb286ed9a9647aaeecc4aa8f26f3ee12d41ec3366c6296eae22"} Oct 30 20:42:40.985997 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:40.985972 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d" Oct 30 20:42:41.088317 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:41.088282 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdba020f-8e2d-40d5-9f78-17970b545e80-kube-api-access-m67x5\" (UniqueName: \"kubernetes.io/projected/cdba020f-8e2d-40d5-9f78-17970b545e80-kube-api-access-m67x5\") pod \"cdba020f-8e2d-40d5-9f78-17970b545e80\" (UID: \"cdba020f-8e2d-40d5-9f78-17970b545e80\") " Oct 30 20:42:41.088551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:41.088328 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdba020f-8e2d-40d5-9f78-17970b545e80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdba020f-8e2d-40d5-9f78-17970b545e80-maas-api-openshift-ca\") pod \"cdba020f-8e2d-40d5-9f78-17970b545e80\" (UID: \"cdba020f-8e2d-40d5-9f78-17970b545e80\") " Oct 30 20:42:41.088551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:41.088498 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdba020f-8e2d-40d5-9f78-17970b545e80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdba020f-8e2d-40d5-9f78-17970b545e80-maas-api-xks-ca\") pod \"cdba020f-8e2d-40d5-9f78-17970b545e80\" (UID: \"cdba020f-8e2d-40d5-9f78-17970b545e80\") " Oct 30 20:42:41.088690 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:41.088664 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdba020f-8e2d-40d5-9f78-17970b545e80-maas-api-openshift-ca" pod "cdba020f-8e2d-40d5-9f78-17970b545e80" (UID: "cdba020f-8e2d-40d5-9f78-17970b545e80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:42:41.090551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:41.090524 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdba020f-8e2d-40d5-9f78-17970b545e80-maas-api-xks-ca" pod "cdba020f-8e2d-40d5-9f78-17970b545e80" (UID: "cdba020f-8e2d-40d5-9f78-17970b545e80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:42:41.090661 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:41.090586 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdba020f-8e2d-40d5-9f78-17970b545e80-kube-api-access-m67x5" pod "cdba020f-8e2d-40d5-9f78-17970b545e80" (UID: "cdba020f-8e2d-40d5-9f78-17970b545e80"). InnerVolumeSpecName "kube-api-access-m67x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:42:41.190243 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:41.190153 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdba020f-8e2d-40d5-9f78-17970b545e80-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:42:41.190243 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:41.190198 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m67x5\" (UniqueName: \"kubernetes.io/projected/cdba020f-8e2d-40d5-9f78-17970b545e80-kube-api-access-m67x5\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:42:41.190243 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:41.190215 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdba020f-8e2d-40d5-9f78-17970b545e80-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:42:41.254877 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:42:41.254841 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:42:41.855253 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:41.855221 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d" Oct 30 20:42:41.855452 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:41.855267 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d" event={"ID":"cdba020f-8e2d-40d5-9f78-17970b545e80","Type":"ContainerDied","Data":"92b0362febfe6bb286ed9a9647aaeecc4aa8f26f3ee12d41ec3366c6296eae22"} Oct 30 20:42:41.855452 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:41.855314 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92b0362febfe6bb286ed9a9647aaeecc4aa8f26f3ee12d41ec3366c6296eae22" Oct 30 20:42:43.763600 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:43.763562 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp"] Oct 30 20:42:43.764010 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:43.763997 5755 cpu_manager.go:414] "containerMap: removing container" podUID="cdba020f-8e2d-40d5-9f78-17970b545e80" containerName="revoke-keys" Oct 30 20:42:43.764010 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:43.764009 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdba020f-8e2d-40d5-9f78-17970b545e80" containerName="revoke-keys" Oct 30 20:42:43.764086 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:43.764082 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdba020f-8e2d-40d5-9f78-17970b545e80" containerName="revoke-keys" Oct 30 20:42:43.767220 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:43.767205 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:43.822124 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:43.822099 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-395022-openshift-default-dockercfg-7nmw8\"" Oct 30 20:42:43.873047 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:43.873013 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp"] Oct 30 20:42:43.915736 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:43.915688 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:43.915906 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:43.915803 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ds42\" (UniqueName: \"kubernetes.io/projected/1e3d6326-166f-47f0-888e-f7a5e8464aa1-kube-api-access-8ds42\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:43.915906 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:43.915852 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:43.915906 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:43.915869 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:43.915906 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:43.915885 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:43.915906 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:43.915906 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:43.916074 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:43.916014 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:43.916074 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:43.916040 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:43.916135 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:43.916072 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.016707 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.016618 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.016707 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.016679 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ds42\" (UniqueName: \"kubernetes.io/projected/1e3d6326-166f-47f0-888e-f7a5e8464aa1-kube-api-access-8ds42\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.016949 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.016719 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.016949 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.016736 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.016949 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.016754 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.016949 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.016783 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.016949 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.016834 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.016949 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.016888 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.017289 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.016993 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.017289 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.017076 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.017289 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.017137 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.017289 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.017231 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.017484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.017374 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.017713 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.017688 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.019212 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.019192 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.019460 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.019442 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.076093 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.076062 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.076240 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.076097 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ds42\" (UniqueName: \"kubernetes.io/projected/1e3d6326-166f-47f0-888e-f7a5e8464aa1-kube-api-access-8ds42\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-vknjp\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.079954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.079927 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:44.514094 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.514064 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp"] Oct 30 20:42:44.515581 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:42:44.515557 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3d6326_166f_47f0_888e_f7a5e8464aa1.slice/crio-101d045a05929d808bed90581fff265cd6da592ece6960ba5313b147be1ecf70 WatchSource:0}: Error finding container 101d045a05929d808bed90581fff265cd6da592ece6960ba5313b147be1ecf70: Status 404 returned error can't find the container with id 101d045a05929d808bed90581fff265cd6da592ece6960ba5313b147be1ecf70 Oct 30 20:42:44.518322 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.518281 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:42:44.518416 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.518378 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:42:44.518468 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.518440 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:42:44.874354 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.874272 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" event={"ID":"1e3d6326-166f-47f0-888e-f7a5e8464aa1","Type":"ContainerStarted","Data":"696e390e9ea0c046828e9c8dbfedc266101bd78975b79f587da363163bd32f94"} Oct 30 20:42:44.874354 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:44.874328 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" event={"ID":"1e3d6326-166f-47f0-888e-f7a5e8464aa1","Type":"ContainerStarted","Data":"101d045a05929d808bed90581fff265cd6da592ece6960ba5313b147be1ecf70"} Oct 30 20:42:45.008243 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:45.008166 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" podStartSLOduration=2.008145353 podStartE2EDuration="2.008145353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:45.006863618 +0000 UTC m=+927.933420558" watchObservedRunningTime="2026-10-30 20:42:45.008145353 +0000 UTC m=+927.934702269" Oct 30 20:42:45.089803 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:45.089769 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:45.094986 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:45.094953 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:42:45.095273 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:45.095240 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" podUID="1e3d6326-166f-47f0-888e-f7a5e8464aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 20:42:46.094415 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:46.092201 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:42:46.094415 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:46.092260 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" podUID="1e3d6326-166f-47f0-888e-f7a5e8464aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 20:42:46.320403 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:46.320356 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:46.333278 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:46.333241 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:46.916858 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:46.916824 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:42:47.094935 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:47.094899 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:42:47.095570 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:47.095537 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" podUID="1e3d6326-166f-47f0-888e-f7a5e8464aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 20:42:48.082854 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:48.082813 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:42:48.083159 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:48.083122 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" podUID="1e3d6326-166f-47f0-888e-f7a5e8464aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 20:42:49.089888 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:49.089849 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:42:49.090545 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:49.090511 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" podUID="1e3d6326-166f-47f0-888e-f7a5e8464aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 20:42:50.085365 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:50.085323 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:42:50.085735 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:50.085702 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" podUID="1e3d6326-166f-47f0-888e-f7a5e8464aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 20:42:51.215011 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:51.214968 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:51.215598 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:51.215574 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:51.216293 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:51.216268 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:42:59.642761 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.642726 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb"] Oct 30 20:42:59.649883 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.649865 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.698243 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.698203 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.698422 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.698249 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.698422 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.698282 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.698422 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.698357 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.698566 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.698471 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89ccbe51-66d9-4fab-836f-50eaff323f79-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.698566 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.698496 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.698566 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.698533 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f8c6\" (UniqueName: \"kubernetes.io/projected/89ccbe51-66d9-4fab-836f-50eaff323f79-kube-api-access-2f8c6\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.698669 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.698651 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.698705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.698682 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.760629 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.760588 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb"] Oct 30 20:42:59.766430 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.766380 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b43c84-openshift-default-dockercfg-xpbj6\"" Oct 30 20:42:59.799855 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.799818 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.800038 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.799863 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.800038 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.799897 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.800038 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.799927 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.800038 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.799989 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89ccbe51-66d9-4fab-836f-50eaff323f79-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.800038 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.800013 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.800301 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.800065 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f8c6\" (UniqueName: \"kubernetes.io/projected/89ccbe51-66d9-4fab-836f-50eaff323f79-kube-api-access-2f8c6\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.800301 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.800155 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.800301 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.800204 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.800608 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.800551 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.800693 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.800600 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.800693 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.800654 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.800826 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.800805 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.800950 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.800931 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89ccbe51-66d9-4fab-836f-50eaff323f79-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.802450 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.802422 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.802583 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.802569 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.859222 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.859188 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.859523 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.859505 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f8c6\" (UniqueName: \"kubernetes.io/projected/89ccbe51-66d9-4fab-836f-50eaff323f79-kube-api-access-2f8c6\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:42:59.963246 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:42:59.963170 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:43:00.390604 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:00.390576 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb"] Oct 30 20:43:00.392278 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:43:00.392254 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ccbe51_66d9_4fab_836f_50eaff323f79.slice/crio-a24abe144fb6ea5a13b440431cc5bf4936f2de1b64c5858ac224da67c283b34e WatchSource:0}: Error finding container a24abe144fb6ea5a13b440431cc5bf4936f2de1b64c5858ac224da67c283b34e: Status 404 returned error can't find the container with id a24abe144fb6ea5a13b440431cc5bf4936f2de1b64c5858ac224da67c283b34e Oct 30 20:43:00.394662 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:00.394609 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:43:00.394757 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:00.394675 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:43:00.394757 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:00.394703 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:43:00.991054 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:00.991016 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" event={"ID":"89ccbe51-66d9-4fab-836f-50eaff323f79","Type":"ContainerStarted","Data":"b177f31361405336a932edfb4d1cf81c86ab52d980e2e59921f6084da07b747d"} Oct 30 20:43:00.991487 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:00.991079 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" event={"ID":"89ccbe51-66d9-4fab-836f-50eaff323f79","Type":"ContainerStarted","Data":"a24abe144fb6ea5a13b440431cc5bf4936f2de1b64c5858ac224da67c283b34e"} Oct 30 20:43:01.209410 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:01.209326 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" podStartSLOduration=2.209308071 podStartE2EDuration="2.209308071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:01.158717835 +0000 UTC m=+944.085274757" watchObservedRunningTime="2026-10-30 20:43:01.209308071 +0000 UTC m=+944.135865000" Oct 30 20:43:01.963757 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:01.963724 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:43:01.965204 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:01.965178 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 20:43:01.965348 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:01.965233 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" podUID="89ccbe51-66d9-4fab-836f-50eaff323f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 20:43:02.094081 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:02.094032 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624"] Oct 30 20:43:02.095050 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:02.095005 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" podUID="26d3f974-97c8-4aea-bc66-563950d17eb9" containerName="istio-proxy" containerID="cri-o://a2eb4f452e5486da588d92ec624f2e23a97ed9b008b3a3d1fad0968d2addb12e" gracePeriod=30 Oct 30 20:43:02.963713 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:02.963674 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 20:43:02.963917 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:02.963723 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" podUID="89ccbe51-66d9-4fab-836f-50eaff323f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 20:43:03.964239 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:03.964203 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 20:43:03.964649 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:03.964257 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" podUID="89ccbe51-66d9-4fab-836f-50eaff323f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 20:43:04.964086 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:04.964047 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 20:43:04.964275 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:04.964108 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" podUID="89ccbe51-66d9-4fab-836f-50eaff323f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 20:43:05.963716 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:05.963680 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 20:43:05.963911 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:05.963739 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" podUID="89ccbe51-66d9-4fab-836f-50eaff323f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 20:43:06.963763 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:06.963725 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 20:43:06.964144 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:06.963787 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" podUID="89ccbe51-66d9-4fab-836f-50eaff323f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 20:43:07.356497 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.356474 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:43:07.472590 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.472560 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-workload-socket\") pod \"26d3f974-97c8-4aea-bc66-563950d17eb9\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " Oct 30 20:43:07.472756 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.472658 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-token\" (UniqueName: \"kubernetes.io/projected/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-token\") pod \"26d3f974-97c8-4aea-bc66-563950d17eb9\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " Oct 30 20:43:07.472756 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.472692 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-envoy\") pod \"26d3f974-97c8-4aea-bc66-563950d17eb9\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " Oct 30 20:43:07.472858 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.472755 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-data\") pod \"26d3f974-97c8-4aea-bc66-563950d17eb9\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " Oct 30 20:43:07.472858 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.472780 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-podinfo\") pod \"26d3f974-97c8-4aea-bc66-563950d17eb9\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " Oct 30 20:43:07.472858 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.472809 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d3f974-97c8-4aea-bc66-563950d17eb9-kube-api-access-ch9pw\" (UniqueName: \"kubernetes.io/projected/26d3f974-97c8-4aea-bc66-563950d17eb9-kube-api-access-ch9pw\") pod \"26d3f974-97c8-4aea-bc66-563950d17eb9\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " Oct 30 20:43:07.473003 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.472863 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-workload-certs\") pod \"26d3f974-97c8-4aea-bc66-563950d17eb9\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " Oct 30 20:43:07.473003 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.472809 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-workload-socket" pod "26d3f974-97c8-4aea-bc66-563950d17eb9" (UID: "26d3f974-97c8-4aea-bc66-563950d17eb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:43:07.473003 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.472923 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d3f974-97c8-4aea-bc66-563950d17eb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d3f974-97c8-4aea-bc66-563950d17eb9-istiod-ca-cert\") pod \"26d3f974-97c8-4aea-bc66-563950d17eb9\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " Oct 30 20:43:07.473110 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.473013 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-credential-socket\") pod \"26d3f974-97c8-4aea-bc66-563950d17eb9\" (UID: \"26d3f974-97c8-4aea-bc66-563950d17eb9\") " Oct 30 20:43:07.473175 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.473119 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-data" pod "26d3f974-97c8-4aea-bc66-563950d17eb9" (UID: "26d3f974-97c8-4aea-bc66-563950d17eb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:43:07.473409 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.473364 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-credential-socket" pod "26d3f974-97c8-4aea-bc66-563950d17eb9" (UID: "26d3f974-97c8-4aea-bc66-563950d17eb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:43:07.473565 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.473459 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:43:07.473565 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.473481 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:43:07.473749 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.473717 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-workload-certs" pod "26d3f974-97c8-4aea-bc66-563950d17eb9" (UID: "26d3f974-97c8-4aea-bc66-563950d17eb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:43:07.473834 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.473807 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d3f974-97c8-4aea-bc66-563950d17eb9-istiod-ca-cert" pod "26d3f974-97c8-4aea-bc66-563950d17eb9" (UID: "26d3f974-97c8-4aea-bc66-563950d17eb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:43:07.475305 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.475283 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-podinfo" pod "26d3f974-97c8-4aea-bc66-563950d17eb9" (UID: "26d3f974-97c8-4aea-bc66-563950d17eb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:43:07.475397 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.475310 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-token" pod "26d3f974-97c8-4aea-bc66-563950d17eb9" (UID: "26d3f974-97c8-4aea-bc66-563950d17eb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:43:07.475807 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.475774 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-envoy" pod "26d3f974-97c8-4aea-bc66-563950d17eb9" (UID: "26d3f974-97c8-4aea-bc66-563950d17eb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:43:07.475908 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.475820 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d3f974-97c8-4aea-bc66-563950d17eb9-kube-api-access-ch9pw" pod "26d3f974-97c8-4aea-bc66-563950d17eb9" (UID: "26d3f974-97c8-4aea-bc66-563950d17eb9"). InnerVolumeSpecName "kube-api-access-ch9pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:43:07.574218 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.574146 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:43:07.574218 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.574173 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:43:07.574218 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.574183 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:43:07.574218 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.574192 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d3f974-97c8-4aea-bc66-563950d17eb9-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:43:07.574218 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.574201 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch9pw\" (UniqueName: \"kubernetes.io/projected/26d3f974-97c8-4aea-bc66-563950d17eb9-kube-api-access-ch9pw\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:43:07.574218 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.574211 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d3f974-97c8-4aea-bc66-563950d17eb9-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:43:07.574218 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.574219 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d3f974-97c8-4aea-bc66-563950d17eb9-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:43:07.968213 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.968188 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:43:07.968672 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.968423 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:43:07.969066 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:07.969051 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:43:08.038797 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:08.038764 5755 generic.go:356] "Generic (PLEG): container finished" podID="26d3f974-97c8-4aea-bc66-563950d17eb9" containerID="a2eb4f452e5486da588d92ec624f2e23a97ed9b008b3a3d1fad0968d2addb12e" exitCode=0 Oct 30 20:43:08.038923 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:08.038808 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" Oct 30 20:43:08.038923 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:08.038851 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" event={"ID":"26d3f974-97c8-4aea-bc66-563950d17eb9","Type":"ContainerDied","Data":"a2eb4f452e5486da588d92ec624f2e23a97ed9b008b3a3d1fad0968d2addb12e"} Oct 30 20:43:08.038923 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:08.038901 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624" event={"ID":"26d3f974-97c8-4aea-bc66-563950d17eb9","Type":"ContainerDied","Data":"92223e273ff3b02adc2e4f5e27fedbacc8e0369f228a2f952f63b28967575baf"} Oct 30 20:43:08.039021 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:08.038928 5755 scope.go:122] "RemoveContainer" containerID="a2eb4f452e5486da588d92ec624f2e23a97ed9b008b3a3d1fad0968d2addb12e" Oct 30 20:43:08.048759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:08.048745 5755 scope.go:122] "RemoveContainer" containerID="a2eb4f452e5486da588d92ec624f2e23a97ed9b008b3a3d1fad0968d2addb12e" Oct 30 20:43:08.049032 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:43:08.049011 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2eb4f452e5486da588d92ec624f2e23a97ed9b008b3a3d1fad0968d2addb12e\": container with ID starting with a2eb4f452e5486da588d92ec624f2e23a97ed9b008b3a3d1fad0968d2addb12e not found: ID does not exist" containerID="a2eb4f452e5486da588d92ec624f2e23a97ed9b008b3a3d1fad0968d2addb12e" Oct 30 20:43:08.049081 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:08.049039 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2eb4f452e5486da588d92ec624f2e23a97ed9b008b3a3d1fad0968d2addb12e"} err="failed to get container status \"a2eb4f452e5486da588d92ec624f2e23a97ed9b008b3a3d1fad0968d2addb12e\": rpc error: code = NotFound desc = could not find container \"a2eb4f452e5486da588d92ec624f2e23a97ed9b008b3a3d1fad0968d2addb12e\": container with ID starting with a2eb4f452e5486da588d92ec624f2e23a97ed9b008b3a3d1fad0968d2addb12e not found: ID does not exist" Oct 30 20:43:08.204116 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:08.204080 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624"] Oct 30 20:43:08.208134 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:08.208107 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fl624"] Oct 30 20:43:09.211838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:09.211808 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d3f974-97c8-4aea-bc66-563950d17eb9" path="/var/lib/kubelet/pods/26d3f974-97c8-4aea-bc66-563950d17eb9/volumes" Oct 30 20:43:11.255188 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:43:11.255153 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:43:15.161647 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.161617 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj"] Oct 30 20:43:15.162041 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.162029 5755 cpu_manager.go:414] "containerMap: removing container" podUID="26d3f974-97c8-4aea-bc66-563950d17eb9" containerName="istio-proxy" Oct 30 20:43:15.162083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.162042 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d3f974-97c8-4aea-bc66-563950d17eb9" containerName="istio-proxy" Oct 30 20:43:15.162129 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.162119 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d3f974-97c8-4aea-bc66-563950d17eb9" containerName="istio-proxy" Oct 30 20:43:15.166818 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.166797 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.220760 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.220733 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b32941-openshift-default-dockercfg-rlxkq\"" Oct 30 20:43:15.259260 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.259227 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bs4d\" (UniqueName: \"kubernetes.io/projected/9847155d-936e-482a-a172-d79a44533b2c-kube-api-access-9bs4d\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.259482 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.259270 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9847155d-936e-482a-a172-d79a44533b2c-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.259482 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.259427 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.259482 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.259453 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.259482 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.259474 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.259643 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.259494 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9847155d-936e-482a-a172-d79a44533b2c-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.259643 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.259596 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.259705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.259649 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.259705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.259692 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9847155d-936e-482a-a172-d79a44533b2c-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.272212 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.272183 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj"] Oct 30 20:43:15.361232 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.361203 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9847155d-936e-482a-a172-d79a44533b2c-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.361488 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.361243 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.361488 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.361265 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.361488 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.361306 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9847155d-936e-482a-a172-d79a44533b2c-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.361488 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.361369 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bs4d\" (UniqueName: \"kubernetes.io/projected/9847155d-936e-482a-a172-d79a44533b2c-kube-api-access-9bs4d\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.361488 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.361417 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9847155d-936e-482a-a172-d79a44533b2c-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.361815 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.361687 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.361815 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.361742 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.361815 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.361802 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.361954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.361823 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.361954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.361863 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.362197 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.362170 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.362322 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.362170 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.362427 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.362401 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9847155d-936e-482a-a172-d79a44533b2c-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.363953 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.363927 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9847155d-936e-482a-a172-d79a44533b2c-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.364079 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.364054 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.420298 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.420226 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bs4d\" (UniqueName: \"kubernetes.io/projected/9847155d-936e-482a-a172-d79a44533b2c-kube-api-access-9bs4d\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.420738 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.420721 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9847155d-936e-482a-a172-d79a44533b2c-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.480334 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.480303 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:15.908958 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.908934 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj"] Oct 30 20:43:15.910340 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:43:15.910311 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9847155d_936e_482a_a172_d79a44533b2c.slice/crio-a3fdf5abdc696ef5dcf6c2eca0c35c50d4d1e1c1156ec4f59949ae38cd7cf731 WatchSource:0}: Error finding container a3fdf5abdc696ef5dcf6c2eca0c35c50d4d1e1c1156ec4f59949ae38cd7cf731: Status 404 returned error can't find the container with id a3fdf5abdc696ef5dcf6c2eca0c35c50d4d1e1c1156ec4f59949ae38cd7cf731 Oct 30 20:43:15.912986 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.912953 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:43:15.913092 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.913030 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:43:15.913092 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:15.913073 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:43:16.085160 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:16.085126 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" event={"ID":"9847155d-936e-482a-a172-d79a44533b2c","Type":"ContainerStarted","Data":"592c2221434b29acdac0e4d52dac188eb1d0f6a31bdfa7dcd8ed17ff88c81722"} Oct 30 20:43:16.085305 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:16.085181 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" event={"ID":"9847155d-936e-482a-a172-d79a44533b2c","Type":"ContainerStarted","Data":"a3fdf5abdc696ef5dcf6c2eca0c35c50d4d1e1c1156ec4f59949ae38cd7cf731"} Oct 30 20:43:16.204532 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:16.204414 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" podStartSLOduration=1.204377817 podStartE2EDuration="1.204377817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:16.203544594 +0000 UTC m=+959.130101531" watchObservedRunningTime="2026-10-30 20:43:16.204377817 +0000 UTC m=+959.130934774" Oct 30 20:43:16.482485 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:16.482448 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:16.486130 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:16.486097 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:43:16.486867 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:16.486827 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" podUID="9847155d-936e-482a-a172-d79a44533b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 20:43:17.485883 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:17.485842 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:43:17.486522 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:17.486487 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" podUID="9847155d-936e-482a-a172-d79a44533b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 20:43:18.486412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:18.483673 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:43:18.486412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:18.483732 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" podUID="9847155d-936e-482a-a172-d79a44533b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 20:43:19.488412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:19.483823 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:43:19.488412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:19.483908 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" podUID="9847155d-936e-482a-a172-d79a44533b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 20:43:20.485881 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:20.485847 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:43:20.486238 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:20.486207 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" podUID="9847155d-936e-482a-a172-d79a44533b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 20:43:21.483417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:21.482687 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:43:21.483417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:21.482747 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" podUID="9847155d-936e-482a-a172-d79a44533b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 20:43:22.480860 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:22.480826 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:43:22.481009 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:22.480875 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" podUID="9847155d-936e-482a-a172-d79a44533b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 20:43:23.485534 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:23.485504 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:23.485968 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:23.485792 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:23.486560 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:43:23.486542 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:43:41.260638 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:43:41.260593 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:44:00.967098 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:00.967018 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n"] Oct 30 20:44:00.970809 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:00.970788 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.041938 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.041909 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n"] Oct 30 20:44:01.052772 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.052741 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-model-cache\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.052946 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.052784 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28mm6\" (UniqueName: \"kubernetes.io/projected/f8dce353-15d0-4501-9315-bdf52c45f3c5-kube-api-access-28mm6\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.052946 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.052816 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-home\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.052946 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.052931 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8dce353-15d0-4501-9315-bdf52c45f3c5-tls-certs\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.053056 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.052984 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-dshm\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.053113 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.053100 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-tmp-dir\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.076068 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.076036 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-386743\"/\"distinct-w5-386743-kserve-self-signed-certs\"" Oct 30 20:44:01.153855 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.153819 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-tmp-dir\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.154024 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.153917 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-model-cache\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.154024 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.153958 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28mm6\" (UniqueName: \"kubernetes.io/projected/f8dce353-15d0-4501-9315-bdf52c45f3c5-kube-api-access-28mm6\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.154113 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.154090 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-home\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.154179 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.154164 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8dce353-15d0-4501-9315-bdf52c45f3c5-tls-certs\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.154225 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.154213 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-dshm\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.154274 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.154248 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-tmp-dir\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.154322 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.154277 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-model-cache\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.154471 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.154452 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-home\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.156641 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.156619 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-dshm\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.156937 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.156915 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8dce353-15d0-4501-9315-bdf52c45f3c5-tls-certs\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.213068 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.213027 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28mm6\" (UniqueName: \"kubernetes.io/projected/f8dce353-15d0-4501-9315-bdf52c45f3c5-kube-api-access-28mm6\") pod \"distinct-w5-386743-kserve-6dffd6bf4f-p5g9n\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.283662 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.283620 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:01.673411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:01.673371 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n"] Oct 30 20:44:01.674880 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:44:01.674853 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8dce353_15d0_4501_9315_bdf52c45f3c5.slice/crio-2e639f6ac6d3179a38e7c5dec1e17c43b1bab42a683ebf542f9b2744bcd6b57b WatchSource:0}: Error finding container 2e639f6ac6d3179a38e7c5dec1e17c43b1bab42a683ebf542f9b2744bcd6b57b: Status 404 returned error can't find the container with id 2e639f6ac6d3179a38e7c5dec1e17c43b1bab42a683ebf542f9b2744bcd6b57b Oct 30 20:44:02.368604 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:02.368570 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" event={"ID":"f8dce353-15d0-4501-9315-bdf52c45f3c5","Type":"ContainerStarted","Data":"d999260c4cdcb302b6a30f0d7e2c0bcaa7e5b8435e8d7988fb3ed90fd63d03ec"} Oct 30 20:44:02.369051 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:02.368612 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" event={"ID":"f8dce353-15d0-4501-9315-bdf52c45f3c5","Type":"ContainerStarted","Data":"2e639f6ac6d3179a38e7c5dec1e17c43b1bab42a683ebf542f9b2744bcd6b57b"} Oct 30 20:44:02.485414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:02.485336 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podStartSLOduration=2.485323353 podStartE2EDuration="2.485323353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:02.485190217 +0000 UTC m=+1005.411747135" watchObservedRunningTime="2026-10-30 20:44:02.485323353 +0000 UTC m=+1005.411880267" Oct 30 20:44:03.284196 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:03.284168 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:11.256400 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:44:11.256343 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:44:11.283794 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:11.283763 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:11.296902 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:11.296878 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:11.430899 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:44:11.430871 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:44:41.254873 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:44:41.254831 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:45:00.186497 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.186464 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7"] Oct 30 20:45:00.191701 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.191681 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" Oct 30 20:45:00.196880 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.196854 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6"] Oct 30 20:45:00.201036 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.201018 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" Oct 30 20:45:00.207511 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.207493 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv"] Oct 30 20:45:00.215466 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.213920 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" Oct 30 20:45:00.236026 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.235997 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4"] Oct 30 20:45:00.239614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.239596 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" Oct 30 20:45:00.254655 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.254634 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:45:00.286257 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.286217 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxkh4\" (UniqueName: \"kubernetes.io/projected/5895ebd0-b52d-4de7-a0d9-699516c25dd3-kube-api-access-rxkh4\") pod \"maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7\" (UID: \"5895ebd0-b52d-4de7-a0d9-699516c25dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" Oct 30 20:45:00.286448 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.286272 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckffs\" (UniqueName: \"kubernetes.io/projected/d25dea36-d864-42c7-a8e0-bc178bec1006-kube-api-access-ckffs\") pod \"maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6\" (UID: \"d25dea36-d864-42c7-a8e0-bc178bec1006\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" Oct 30 20:45:00.286448 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.286364 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53b00f5-d023-4ea3-a163-f588849085ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4\" (UID: \"e53b00f5-d023-4ea3-a163-f588849085ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" Oct 30 20:45:00.286526 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.286485 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr8jg\" (UniqueName: \"kubernetes.io/projected/2d6bc14f-d81b-4b3e-80fb-28be2cef44bb-kube-api-access-cr8jg\") pod \"maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv\" (UID: \"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" Oct 30 20:45:00.286526 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.286516 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6bc14f-d81b-4b3e-80fb-28be2cef44bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv\" (UID: \"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" Oct 30 20:45:00.286589 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.286572 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25dea36-d864-42c7-a8e0-bc178bec1006-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6\" (UID: \"d25dea36-d864-42c7-a8e0-bc178bec1006\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" Oct 30 20:45:00.286629 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.286607 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5895ebd0-b52d-4de7-a0d9-699516c25dd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7\" (UID: \"5895ebd0-b52d-4de7-a0d9-699516c25dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" Oct 30 20:45:00.286678 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.286670 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t24rm\" (UniqueName: \"kubernetes.io/projected/e53b00f5-d023-4ea3-a163-f588849085ac-kube-api-access-t24rm\") pod \"maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4\" (UID: \"e53b00f5-d023-4ea3-a163-f588849085ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" Oct 30 20:45:00.291333 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.291299 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7"] Oct 30 20:45:00.332729 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.332692 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6"] Oct 30 20:45:00.337163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.337100 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4"] Oct 30 20:45:00.340148 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.340121 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv"] Oct 30 20:45:00.388227 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.388186 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t24rm\" (UniqueName: \"kubernetes.io/projected/e53b00f5-d023-4ea3-a163-f588849085ac-kube-api-access-t24rm\") pod \"maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4\" (UID: \"e53b00f5-d023-4ea3-a163-f588849085ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" Oct 30 20:45:00.388486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.388276 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxkh4\" (UniqueName: \"kubernetes.io/projected/5895ebd0-b52d-4de7-a0d9-699516c25dd3-kube-api-access-rxkh4\") pod \"maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7\" (UID: \"5895ebd0-b52d-4de7-a0d9-699516c25dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" Oct 30 20:45:00.388486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.388322 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckffs\" (UniqueName: \"kubernetes.io/projected/d25dea36-d864-42c7-a8e0-bc178bec1006-kube-api-access-ckffs\") pod \"maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6\" (UID: \"d25dea36-d864-42c7-a8e0-bc178bec1006\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" Oct 30 20:45:00.388486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.388355 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53b00f5-d023-4ea3-a163-f588849085ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4\" (UID: \"e53b00f5-d023-4ea3-a163-f588849085ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" Oct 30 20:45:00.388486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.388453 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8jg\" (UniqueName: \"kubernetes.io/projected/2d6bc14f-d81b-4b3e-80fb-28be2cef44bb-kube-api-access-cr8jg\") pod \"maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv\" (UID: \"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" Oct 30 20:45:00.388486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.388485 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6bc14f-d81b-4b3e-80fb-28be2cef44bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv\" (UID: \"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" Oct 30 20:45:00.388764 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.388512 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25dea36-d864-42c7-a8e0-bc178bec1006-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6\" (UID: \"d25dea36-d864-42c7-a8e0-bc178bec1006\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" Oct 30 20:45:00.388764 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.388558 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5895ebd0-b52d-4de7-a0d9-699516c25dd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7\" (UID: \"5895ebd0-b52d-4de7-a0d9-699516c25dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" Oct 30 20:45:00.389354 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.389297 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6bc14f-d81b-4b3e-80fb-28be2cef44bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv\" (UID: \"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" Oct 30 20:45:00.389549 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.389352 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53b00f5-d023-4ea3-a163-f588849085ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4\" (UID: \"e53b00f5-d023-4ea3-a163-f588849085ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" Oct 30 20:45:00.389549 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.389506 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25dea36-d864-42c7-a8e0-bc178bec1006-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6\" (UID: \"d25dea36-d864-42c7-a8e0-bc178bec1006\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" Oct 30 20:45:00.389801 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.389783 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5895ebd0-b52d-4de7-a0d9-699516c25dd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7\" (UID: \"5895ebd0-b52d-4de7-a0d9-699516c25dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" Oct 30 20:45:00.447600 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.447527 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t24rm\" (UniqueName: \"kubernetes.io/projected/e53b00f5-d023-4ea3-a163-f588849085ac-kube-api-access-t24rm\") pod \"maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4\" (UID: \"e53b00f5-d023-4ea3-a163-f588849085ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" Oct 30 20:45:00.447777 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.447752 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxkh4\" (UniqueName: \"kubernetes.io/projected/5895ebd0-b52d-4de7-a0d9-699516c25dd3-kube-api-access-rxkh4\") pod \"maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7\" (UID: \"5895ebd0-b52d-4de7-a0d9-699516c25dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" Oct 30 20:45:00.448050 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.448030 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr8jg\" (UniqueName: \"kubernetes.io/projected/2d6bc14f-d81b-4b3e-80fb-28be2cef44bb-kube-api-access-cr8jg\") pod \"maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv\" (UID: \"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" Oct 30 20:45:00.448207 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.448186 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckffs\" (UniqueName: \"kubernetes.io/projected/d25dea36-d864-42c7-a8e0-bc178bec1006-kube-api-access-ckffs\") pod \"maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6\" (UID: \"d25dea36-d864-42c7-a8e0-bc178bec1006\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" Oct 30 20:45:00.506187 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.506156 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" Oct 30 20:45:00.519548 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.519527 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" Oct 30 20:45:00.529263 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.529239 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" Oct 30 20:45:00.552712 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.552676 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" Oct 30 20:45:00.896954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.896932 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7"] Oct 30 20:45:00.897944 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:45:00.897917 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5895ebd0_b52d_4de7_a0d9_699516c25dd3.slice/crio-9c2f7bcbfe573fb125104f2319236ad09d52dfa74738cc7694919ecce80254d6 WatchSource:0}: Error finding container 9c2f7bcbfe573fb125104f2319236ad09d52dfa74738cc7694919ecce80254d6: Status 404 returned error can't find the container with id 9c2f7bcbfe573fb125104f2319236ad09d52dfa74738cc7694919ecce80254d6 Oct 30 20:45:00.946207 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.946177 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6"] Oct 30 20:45:00.947877 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:45:00.947845 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25dea36_d864_42c7_a8e0_bc178bec1006.slice/crio-804e173b9078d6068de81b262832f53c90e7abbcbc03e821674b3504ba7c28a3 WatchSource:0}: Error finding container 804e173b9078d6068de81b262832f53c90e7abbcbc03e821674b3504ba7c28a3: Status 404 returned error can't find the container with id 804e173b9078d6068de81b262832f53c90e7abbcbc03e821674b3504ba7c28a3 Oct 30 20:45:00.949137 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:00.949112 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv"] Oct 30 20:45:00.951011 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:45:00.950987 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6bc14f_d81b_4b3e_80fb_28be2cef44bb.slice/crio-5405eacb74d3d6777a4b209089ed05db944d9aa8b67efe0dc52643790c766eee WatchSource:0}: Error finding container 5405eacb74d3d6777a4b209089ed05db944d9aa8b67efe0dc52643790c766eee: Status 404 returned error can't find the container with id 5405eacb74d3d6777a4b209089ed05db944d9aa8b67efe0dc52643790c766eee Oct 30 20:45:01.000834 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.000798 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf"] Oct 30 20:45:01.006567 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.006543 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4"] Oct 30 20:45:01.006679 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.006659 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.009353 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:45:01.009325 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53b00f5_d023_4ea3_a163_f588849085ac.slice/crio-6c926312390ac79c787475bfa4c2ce97836023124407956059ce3e79d2c7f73f WatchSource:0}: Error finding container 6c926312390ac79c787475bfa4c2ce97836023124407956059ce3e79d2c7f73f: Status 404 returned error can't find the container with id 6c926312390ac79c787475bfa4c2ce97836023124407956059ce3e79d2c7f73f Oct 30 20:45:01.070226 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.070192 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf"] Oct 30 20:45:01.098458 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.098422 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.098670 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.098470 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.098670 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.098498 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.098811 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.098666 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg2f5\" (UniqueName: \"kubernetes.io/projected/3d477ecb-658f-4ce9-af37-3ca9f557b889-kube-api-access-xg2f5\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.098877 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.098810 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.098949 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.098922 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.099001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.098973 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d477ecb-658f-4ce9-af37-3ca9f557b889-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.099115 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.099097 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.099175 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.099142 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.199990 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.199959 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.200486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.199996 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.200486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.200017 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.200486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.200078 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg2f5\" (UniqueName: \"kubernetes.io/projected/3d477ecb-658f-4ce9-af37-3ca9f557b889-kube-api-access-xg2f5\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.200486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.200144 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.200486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.200202 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.200486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.200235 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d477ecb-658f-4ce9-af37-3ca9f557b889-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.200486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.200303 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.200486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.200338 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.200486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.200453 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.201409 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.200709 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.201409 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.200740 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.201409 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.201060 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d477ecb-658f-4ce9-af37-3ca9f557b889-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.201409 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.201167 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.202821 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.202794 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.202942 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.202925 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.258510 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.258473 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.258664 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.258546 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg2f5\" (UniqueName: \"kubernetes.io/projected/3d477ecb-658f-4ce9-af37-3ca9f557b889-kube-api-access-xg2f5\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.325769 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.325734 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:01.736568 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.736468 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" event={"ID":"e53b00f5-d023-4ea3-a163-f588849085ac","Type":"ContainerStarted","Data":"49951085c085ea864ddbe7b0f258f89b2956b76e265f83d998bd0ab5ab24f5e5"} Oct 30 20:45:01.736568 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.736537 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" event={"ID":"e53b00f5-d023-4ea3-a163-f588849085ac","Type":"ContainerStarted","Data":"6c926312390ac79c787475bfa4c2ce97836023124407956059ce3e79d2c7f73f"} Oct 30 20:45:01.738224 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.738195 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" event={"ID":"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb","Type":"ContainerStarted","Data":"8a3178939d7d27599ca111e19b3fa122361a477b1d7e29f7a074abec93214e50"} Oct 30 20:45:01.738358 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.738244 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" event={"ID":"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb","Type":"ContainerStarted","Data":"5405eacb74d3d6777a4b209089ed05db944d9aa8b67efe0dc52643790c766eee"} Oct 30 20:45:01.739758 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.739736 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" event={"ID":"5895ebd0-b52d-4de7-a0d9-699516c25dd3","Type":"ContainerStarted","Data":"22e14ce782e9a2965e839ef47e6321d4177c4b394c6026d7421a083e70f30de7"} Oct 30 20:45:01.739853 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.739779 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" event={"ID":"5895ebd0-b52d-4de7-a0d9-699516c25dd3","Type":"ContainerStarted","Data":"9c2f7bcbfe573fb125104f2319236ad09d52dfa74738cc7694919ecce80254d6"} Oct 30 20:45:01.741712 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.741693 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" event={"ID":"d25dea36-d864-42c7-a8e0-bc178bec1006","Type":"ContainerStarted","Data":"b728f424feea81158f6ccee30573d2aba32822731642811be891dbaf65c591f9"} Oct 30 20:45:01.741815 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.741726 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" event={"ID":"d25dea36-d864-42c7-a8e0-bc178bec1006","Type":"ContainerStarted","Data":"804e173b9078d6068de81b262832f53c90e7abbcbc03e821674b3504ba7c28a3"} Oct 30 20:45:01.852677 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.852647 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf"] Oct 30 20:45:01.853090 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.853043 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" podStartSLOduration=1.853028459 podStartE2EDuration="1.853028459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:01.850040354 +0000 UTC m=+1064.776597289" watchObservedRunningTime="2026-10-30 20:45:01.853028459 +0000 UTC m=+1064.779585374" Oct 30 20:45:01.854446 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:45:01.854422 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d477ecb_658f_4ce9_af37_3ca9f557b889.slice/crio-7d4c50f693c044a498193a003662fe258e835f15f21914c381ac6b9672572b30 WatchSource:0}: Error finding container 7d4c50f693c044a498193a003662fe258e835f15f21914c381ac6b9672572b30: Status 404 returned error can't find the container with id 7d4c50f693c044a498193a003662fe258e835f15f21914c381ac6b9672572b30 Oct 30 20:45:01.856858 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.856827 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:45:01.856924 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.856891 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:45:01.856970 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.856923 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:45:01.966764 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:01.966707 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" podStartSLOduration=1.966694486 podStartE2EDuration="1.966694486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:01.963571515 +0000 UTC m=+1064.890128431" watchObservedRunningTime="2026-10-30 20:45:01.966694486 +0000 UTC m=+1064.893251400" Oct 30 20:45:02.079493 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:02.079444 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" podStartSLOduration=2.079429932 podStartE2EDuration="2.079429932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:02.077751178 +0000 UTC m=+1065.004308105" watchObservedRunningTime="2026-10-30 20:45:02.079429932 +0000 UTC m=+1065.005986879" Oct 30 20:45:02.194547 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:02.194486 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" podStartSLOduration=2.194465147 podStartE2EDuration="2.194465147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:02.194035489 +0000 UTC m=+1065.120592420" watchObservedRunningTime="2026-10-30 20:45:02.194465147 +0000 UTC m=+1065.121022063" Oct 30 20:45:02.749957 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:02.749913 5755 generic.go:356] "Generic (PLEG): container finished" podID="2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" containerID="8a3178939d7d27599ca111e19b3fa122361a477b1d7e29f7a074abec93214e50" exitCode=60 Oct 30 20:45:02.750435 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:02.750006 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" event={"ID":"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb","Type":"ContainerDied","Data":"8a3178939d7d27599ca111e19b3fa122361a477b1d7e29f7a074abec93214e50"} Oct 30 20:45:02.750435 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:02.750269 5755 scope.go:122] "RemoveContainer" containerID="8a3178939d7d27599ca111e19b3fa122361a477b1d7e29f7a074abec93214e50" Oct 30 20:45:02.752580 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:02.752551 5755 generic.go:356] "Generic (PLEG): container finished" podID="5895ebd0-b52d-4de7-a0d9-699516c25dd3" containerID="22e14ce782e9a2965e839ef47e6321d4177c4b394c6026d7421a083e70f30de7" exitCode=60 Oct 30 20:45:02.752700 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:02.752661 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" event={"ID":"5895ebd0-b52d-4de7-a0d9-699516c25dd3","Type":"ContainerDied","Data":"22e14ce782e9a2965e839ef47e6321d4177c4b394c6026d7421a083e70f30de7"} Oct 30 20:45:02.752827 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:02.752812 5755 scope.go:122] "RemoveContainer" containerID="22e14ce782e9a2965e839ef47e6321d4177c4b394c6026d7421a083e70f30de7" Oct 30 20:45:02.754284 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:02.754263 5755 generic.go:356] "Generic (PLEG): container finished" podID="d25dea36-d864-42c7-a8e0-bc178bec1006" containerID="b728f424feea81158f6ccee30573d2aba32822731642811be891dbaf65c591f9" exitCode=60 Oct 30 20:45:02.754363 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:02.754288 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" event={"ID":"d25dea36-d864-42c7-a8e0-bc178bec1006","Type":"ContainerDied","Data":"b728f424feea81158f6ccee30573d2aba32822731642811be891dbaf65c591f9"} Oct 30 20:45:02.754585 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:02.754570 5755 scope.go:122] "RemoveContainer" containerID="b728f424feea81158f6ccee30573d2aba32822731642811be891dbaf65c591f9" Oct 30 20:45:02.756128 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:02.756112 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" event={"ID":"3d477ecb-658f-4ce9-af37-3ca9f557b889","Type":"ContainerStarted","Data":"a9ce23bdfddc5d30b1ac256e5fea02f7df0bc45065fc2e561220e30ab209efb4"} Oct 30 20:45:02.756190 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:02.756149 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" event={"ID":"3d477ecb-658f-4ce9-af37-3ca9f557b889","Type":"ContainerStarted","Data":"7d4c50f693c044a498193a003662fe258e835f15f21914c381ac6b9672572b30"} Oct 30 20:45:02.757524 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:02.757502 5755 generic.go:356] "Generic (PLEG): container finished" podID="e53b00f5-d023-4ea3-a163-f588849085ac" containerID="49951085c085ea864ddbe7b0f258f89b2956b76e265f83d998bd0ab5ab24f5e5" exitCode=60 Oct 30 20:45:02.757608 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:02.757527 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" event={"ID":"e53b00f5-d023-4ea3-a163-f588849085ac","Type":"ContainerDied","Data":"49951085c085ea864ddbe7b0f258f89b2956b76e265f83d998bd0ab5ab24f5e5"} Oct 30 20:45:02.757785 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:02.757769 5755 scope.go:122] "RemoveContainer" containerID="49951085c085ea864ddbe7b0f258f89b2956b76e265f83d998bd0ab5ab24f5e5" Oct 30 20:45:03.239053 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:03.239001 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" podStartSLOduration=3.23898792 podStartE2EDuration="3.23898792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:03.117100635 +0000 UTC m=+1066.043657550" watchObservedRunningTime="2026-10-30 20:45:03.23898792 +0000 UTC m=+1066.165544835" Oct 30 20:45:03.326400 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:03.326360 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:03.328061 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:03.328039 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:45:03.328151 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:03.328078 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" podUID="3d477ecb-658f-4ce9-af37-3ca9f557b889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 20:45:03.766890 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:03.766841 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" event={"ID":"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb","Type":"ContainerStarted","Data":"47c9a016b5c6ab68999f3d255904a52becc9cdfaadad4c5097d4fe215eb3c4da"} Oct 30 20:45:03.768854 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:03.768828 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" event={"ID":"5895ebd0-b52d-4de7-a0d9-699516c25dd3","Type":"ContainerStarted","Data":"2302ecf1107d66baad4b9d0a31f604c06d2d57d46eb2fe2604f39db1bc1b4d56"} Oct 30 20:45:03.797300 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:03.797253 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" event={"ID":"d25dea36-d864-42c7-a8e0-bc178bec1006","Type":"ContainerStarted","Data":"07c24fa8c8ec6da3aa382a8ca21ef85444f43834098d7482ea37f9c7784a656d"} Oct 30 20:45:03.800079 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:03.800048 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" event={"ID":"e53b00f5-d023-4ea3-a163-f588849085ac","Type":"ContainerStarted","Data":"fad8c8c2c59e959c7f850c18f3bb07995cf34ef83873b8a1b752b7825ee6fcea"} Oct 30 20:45:04.327028 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.326951 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:45:04.327028 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.327000 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" podUID="3d477ecb-658f-4ce9-af37-3ca9f557b889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 20:45:04.808708 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.808668 5755 generic.go:356] "Generic (PLEG): container finished" podID="2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" containerID="47c9a016b5c6ab68999f3d255904a52becc9cdfaadad4c5097d4fe215eb3c4da" exitCode=60 Oct 30 20:45:04.809151 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.808752 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" event={"ID":"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb","Type":"ContainerDied","Data":"47c9a016b5c6ab68999f3d255904a52becc9cdfaadad4c5097d4fe215eb3c4da"} Oct 30 20:45:04.809151 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.808814 5755 scope.go:122] "RemoveContainer" containerID="8a3178939d7d27599ca111e19b3fa122361a477b1d7e29f7a074abec93214e50" Oct 30 20:45:04.809151 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.809050 5755 scope.go:122] "RemoveContainer" containerID="47c9a016b5c6ab68999f3d255904a52becc9cdfaadad4c5097d4fe215eb3c4da" Oct 30 20:45:04.809353 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:45:04.809319 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv_odh-ai-gateway-infra(2d6bc14f-d81b-4b3e-80fb-28be2cef44bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" podUID="2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" Oct 30 20:45:04.810872 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.810852 5755 generic.go:356] "Generic (PLEG): container finished" podID="5895ebd0-b52d-4de7-a0d9-699516c25dd3" containerID="2302ecf1107d66baad4b9d0a31f604c06d2d57d46eb2fe2604f39db1bc1b4d56" exitCode=60 Oct 30 20:45:04.810963 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.810929 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" event={"ID":"5895ebd0-b52d-4de7-a0d9-699516c25dd3","Type":"ContainerDied","Data":"2302ecf1107d66baad4b9d0a31f604c06d2d57d46eb2fe2604f39db1bc1b4d56"} Oct 30 20:45:04.811145 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.811131 5755 scope.go:122] "RemoveContainer" containerID="2302ecf1107d66baad4b9d0a31f604c06d2d57d46eb2fe2604f39db1bc1b4d56" Oct 30 20:45:04.811424 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:45:04.811381 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7_odh-ai-gateway-infra(5895ebd0-b52d-4de7-a0d9-699516c25dd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" podUID="5895ebd0-b52d-4de7-a0d9-699516c25dd3" Oct 30 20:45:04.812780 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.812751 5755 generic.go:356] "Generic (PLEG): container finished" podID="d25dea36-d864-42c7-a8e0-bc178bec1006" containerID="07c24fa8c8ec6da3aa382a8ca21ef85444f43834098d7482ea37f9c7784a656d" exitCode=60 Oct 30 20:45:04.812900 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.812877 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" event={"ID":"d25dea36-d864-42c7-a8e0-bc178bec1006","Type":"ContainerDied","Data":"07c24fa8c8ec6da3aa382a8ca21ef85444f43834098d7482ea37f9c7784a656d"} Oct 30 20:45:04.813040 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.813023 5755 scope.go:122] "RemoveContainer" containerID="07c24fa8c8ec6da3aa382a8ca21ef85444f43834098d7482ea37f9c7784a656d" Oct 30 20:45:04.813283 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:45:04.813261 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6_odh-ai-gateway-infra(d25dea36-d864-42c7-a8e0-bc178bec1006)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" podUID="d25dea36-d864-42c7-a8e0-bc178bec1006" Oct 30 20:45:04.814978 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.814960 5755 generic.go:356] "Generic (PLEG): container finished" podID="e53b00f5-d023-4ea3-a163-f588849085ac" containerID="fad8c8c2c59e959c7f850c18f3bb07995cf34ef83873b8a1b752b7825ee6fcea" exitCode=60 Oct 30 20:45:04.815068 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.815023 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" event={"ID":"e53b00f5-d023-4ea3-a163-f588849085ac","Type":"ContainerDied","Data":"fad8c8c2c59e959c7f850c18f3bb07995cf34ef83873b8a1b752b7825ee6fcea"} Oct 30 20:45:04.815232 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.815221 5755 scope.go:122] "RemoveContainer" containerID="fad8c8c2c59e959c7f850c18f3bb07995cf34ef83873b8a1b752b7825ee6fcea" Oct 30 20:45:04.815503 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:45:04.815487 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4_odh-ai-gateway-infra(e53b00f5-d023-4ea3-a163-f588849085ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" podUID="e53b00f5-d023-4ea3-a163-f588849085ac" Oct 30 20:45:04.824953 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.824936 5755 scope.go:122] "RemoveContainer" containerID="22e14ce782e9a2965e839ef47e6321d4177c4b394c6026d7421a083e70f30de7" Oct 30 20:45:04.837806 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.837783 5755 scope.go:122] "RemoveContainer" containerID="b728f424feea81158f6ccee30573d2aba32822731642811be891dbaf65c591f9" Oct 30 20:45:04.850504 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:04.850481 5755 scope.go:122] "RemoveContainer" containerID="49951085c085ea864ddbe7b0f258f89b2956b76e265f83d998bd0ab5ab24f5e5" Oct 30 20:45:05.326970 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:05.326935 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:45:05.327158 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:05.326991 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" podUID="3d477ecb-658f-4ce9-af37-3ca9f557b889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 20:45:05.824028 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:05.823992 5755 scope.go:122] "RemoveContainer" containerID="2302ecf1107d66baad4b9d0a31f604c06d2d57d46eb2fe2604f39db1bc1b4d56" Oct 30 20:45:05.824540 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:45:05.824296 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7_odh-ai-gateway-infra(5895ebd0-b52d-4de7-a0d9-699516c25dd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" podUID="5895ebd0-b52d-4de7-a0d9-699516c25dd3" Oct 30 20:45:05.825835 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:05.825809 5755 scope.go:122] "RemoveContainer" containerID="07c24fa8c8ec6da3aa382a8ca21ef85444f43834098d7482ea37f9c7784a656d" Oct 30 20:45:05.826080 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:45:05.826060 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6_odh-ai-gateway-infra(d25dea36-d864-42c7-a8e0-bc178bec1006)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" podUID="d25dea36-d864-42c7-a8e0-bc178bec1006" Oct 30 20:45:05.827520 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:05.827504 5755 scope.go:122] "RemoveContainer" containerID="fad8c8c2c59e959c7f850c18f3bb07995cf34ef83873b8a1b752b7825ee6fcea" Oct 30 20:45:05.827701 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:45:05.827681 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4_odh-ai-gateway-infra(e53b00f5-d023-4ea3-a163-f588849085ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" podUID="e53b00f5-d023-4ea3-a163-f588849085ac" Oct 30 20:45:05.829116 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:05.829099 5755 scope.go:122] "RemoveContainer" containerID="47c9a016b5c6ab68999f3d255904a52becc9cdfaadad4c5097d4fe215eb3c4da" Oct 30 20:45:05.829317 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:45:05.829298 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv_odh-ai-gateway-infra(2d6bc14f-d81b-4b3e-80fb-28be2cef44bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" podUID="2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" Oct 30 20:45:06.326501 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:06.326469 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:45:06.326689 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:06.326518 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" podUID="3d477ecb-658f-4ce9-af37-3ca9f557b889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 20:45:07.326305 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:07.326272 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:45:07.326747 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:07.326329 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" podUID="3d477ecb-658f-4ce9-af37-3ca9f557b889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 20:45:08.111875 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.111843 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59"] Oct 30 20:45:08.116657 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.116635 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.186258 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.186217 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-dshm\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.186258 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.186251 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-model-cache\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.186524 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.186364 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-home\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.186524 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.186458 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a305512d-f7b3-4b29-b72e-22115f7c8b0c-tls-certs\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.186524 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.186491 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-tmp-dir\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.186524 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.186507 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjktn\" (UniqueName: \"kubernetes.io/projected/a305512d-f7b3-4b29-b72e-22115f7c8b0c-kube-api-access-vjktn\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.219427 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.219368 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59"] Oct 30 20:45:08.221061 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.221041 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f6ebf5\"/\"distinct-2-w3-f6ebf5-kserve-self-signed-certs\"" Oct 30 20:45:08.288091 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.288055 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-home\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.288091 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.288090 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a305512d-f7b3-4b29-b72e-22115f7c8b0c-tls-certs\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.288324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.288117 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-tmp-dir\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.288324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.288144 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjktn\" (UniqueName: \"kubernetes.io/projected/a305512d-f7b3-4b29-b72e-22115f7c8b0c-kube-api-access-vjktn\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.288487 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.288326 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-dshm\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.288487 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.288354 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-model-cache\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.288620 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.288516 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-home\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.288620 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.288588 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-tmp-dir\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.288767 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.288745 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-model-cache\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.290868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.290842 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-dshm\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.291008 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.290904 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a305512d-f7b3-4b29-b72e-22115f7c8b0c-tls-certs\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.326685 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.326654 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:45:08.327119 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.326711 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" podUID="3d477ecb-658f-4ce9-af37-3ca9f557b889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 20:45:08.348104 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.348074 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjktn\" (UniqueName: \"kubernetes.io/projected/a305512d-f7b3-4b29-b72e-22115f7c8b0c-kube-api-access-vjktn\") pod \"distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.431321 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.431223 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:08.824543 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.824513 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59"] Oct 30 20:45:08.826660 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:45:08.826632 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda305512d_f7b3_4b29_b72e_22115f7c8b0c.slice/crio-7f7a82ab5d2d7aec087c756561bd8dc54a679e1f80760a4549fc7cb14fc1e596 WatchSource:0}: Error finding container 7f7a82ab5d2d7aec087c756561bd8dc54a679e1f80760a4549fc7cb14fc1e596: Status 404 returned error can't find the container with id 7f7a82ab5d2d7aec087c756561bd8dc54a679e1f80760a4549fc7cb14fc1e596 Oct 30 20:45:08.849396 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:08.849353 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" event={"ID":"a305512d-f7b3-4b29-b72e-22115f7c8b0c","Type":"ContainerStarted","Data":"7f7a82ab5d2d7aec087c756561bd8dc54a679e1f80760a4549fc7cb14fc1e596"} Oct 30 20:45:09.417447 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:09.417412 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:09.417963 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:09.417851 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:09.419043 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:09.419020 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:45:09.856621 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:09.856585 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" event={"ID":"a305512d-f7b3-4b29-b72e-22115f7c8b0c","Type":"ContainerStarted","Data":"dcd8ceaf26047b1879641285c433580f16f53e2347eb0b1805fa50c85966986c"} Oct 30 20:45:09.974237 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:09.974182 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podStartSLOduration=1.97416762 podStartE2EDuration="1.97416762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:09.971839534 +0000 UTC m=+1072.898396454" watchObservedRunningTime="2026-10-30 20:45:09.97416762 +0000 UTC m=+1072.900724535" Oct 30 20:45:10.431494 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:10.431462 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:11.255372 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:45:11.255339 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:45:15.205192 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:15.205159 5755 scope.go:122] "RemoveContainer" containerID="2302ecf1107d66baad4b9d0a31f604c06d2d57d46eb2fe2604f39db1bc1b4d56" Oct 30 20:45:15.891424 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:15.891331 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" event={"ID":"5895ebd0-b52d-4de7-a0d9-699516c25dd3","Type":"ContainerStarted","Data":"d22ea9f8457cee0e0891f4c082547555d00fc8506f34cf07adf7cd5424982144"} Oct 30 20:45:16.205101 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:16.205018 5755 scope.go:122] "RemoveContainer" containerID="fad8c8c2c59e959c7f850c18f3bb07995cf34ef83873b8a1b752b7825ee6fcea" Oct 30 20:45:16.899139 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:16.899047 5755 generic.go:356] "Generic (PLEG): container finished" podID="5895ebd0-b52d-4de7-a0d9-699516c25dd3" containerID="d22ea9f8457cee0e0891f4c082547555d00fc8506f34cf07adf7cd5424982144" exitCode=60 Oct 30 20:45:16.899139 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:16.899115 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" event={"ID":"5895ebd0-b52d-4de7-a0d9-699516c25dd3","Type":"ContainerDied","Data":"d22ea9f8457cee0e0891f4c082547555d00fc8506f34cf07adf7cd5424982144"} Oct 30 20:45:16.899743 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:16.899180 5755 scope.go:122] "RemoveContainer" containerID="2302ecf1107d66baad4b9d0a31f604c06d2d57d46eb2fe2604f39db1bc1b4d56" Oct 30 20:45:16.899743 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:16.899602 5755 scope.go:122] "RemoveContainer" containerID="d22ea9f8457cee0e0891f4c082547555d00fc8506f34cf07adf7cd5424982144" Oct 30 20:45:16.899914 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:45:16.899894 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7_odh-ai-gateway-infra(5895ebd0-b52d-4de7-a0d9-699516c25dd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" podUID="5895ebd0-b52d-4de7-a0d9-699516c25dd3" Oct 30 20:45:16.901224 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:16.901203 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" event={"ID":"e53b00f5-d023-4ea3-a163-f588849085ac","Type":"ContainerStarted","Data":"222bd21dc673ff8588909b61ca959c55b73cc1f120150674383e886ee47ad7a7"} Oct 30 20:45:17.026869 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:17.026837 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7"] Oct 30 20:45:17.208298 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:17.208230 5755 scope.go:122] "RemoveContainer" containerID="07c24fa8c8ec6da3aa382a8ca21ef85444f43834098d7482ea37f9c7784a656d" Oct 30 20:45:17.909251 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:17.909174 5755 generic.go:356] "Generic (PLEG): container finished" podID="e53b00f5-d023-4ea3-a163-f588849085ac" containerID="222bd21dc673ff8588909b61ca959c55b73cc1f120150674383e886ee47ad7a7" exitCode=60 Oct 30 20:45:17.909715 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:17.909253 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" event={"ID":"e53b00f5-d023-4ea3-a163-f588849085ac","Type":"ContainerDied","Data":"222bd21dc673ff8588909b61ca959c55b73cc1f120150674383e886ee47ad7a7"} Oct 30 20:45:17.909715 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:17.909354 5755 scope.go:122] "RemoveContainer" containerID="fad8c8c2c59e959c7f850c18f3bb07995cf34ef83873b8a1b752b7825ee6fcea" Oct 30 20:45:17.909842 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:17.909737 5755 scope.go:122] "RemoveContainer" containerID="222bd21dc673ff8588909b61ca959c55b73cc1f120150674383e886ee47ad7a7" Oct 30 20:45:17.910044 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:45:17.910020 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4_odh-ai-gateway-infra(e53b00f5-d023-4ea3-a163-f588849085ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" podUID="e53b00f5-d023-4ea3-a163-f588849085ac" Oct 30 20:45:17.913182 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:17.913153 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" event={"ID":"d25dea36-d864-42c7-a8e0-bc178bec1006","Type":"ContainerStarted","Data":"791c7bd442b1f4a850396f13524db310400f2397710e1649b489cc71f15b1cd8"} Oct 30 20:45:18.047823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.047796 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" Oct 30 20:45:18.145184 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.145153 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4"] Oct 30 20:45:18.196380 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.196299 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5895ebd0-b52d-4de7-a0d9-699516c25dd3-kube-api-access-rxkh4\" (UniqueName: \"kubernetes.io/projected/5895ebd0-b52d-4de7-a0d9-699516c25dd3-kube-api-access-rxkh4\") pod \"5895ebd0-b52d-4de7-a0d9-699516c25dd3\" (UID: \"5895ebd0-b52d-4de7-a0d9-699516c25dd3\") " Oct 30 20:45:18.196380 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.196365 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5895ebd0-b52d-4de7-a0d9-699516c25dd3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5895ebd0-b52d-4de7-a0d9-699516c25dd3-maas-api-service-ca\") pod \"5895ebd0-b52d-4de7-a0d9-699516c25dd3\" (UID: \"5895ebd0-b52d-4de7-a0d9-699516c25dd3\") " Oct 30 20:45:18.196766 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.196739 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5895ebd0-b52d-4de7-a0d9-699516c25dd3-maas-api-service-ca" pod "5895ebd0-b52d-4de7-a0d9-699516c25dd3" (UID: "5895ebd0-b52d-4de7-a0d9-699516c25dd3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:45:18.198634 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.198614 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5895ebd0-b52d-4de7-a0d9-699516c25dd3-kube-api-access-rxkh4" pod "5895ebd0-b52d-4de7-a0d9-699516c25dd3" (UID: "5895ebd0-b52d-4de7-a0d9-699516c25dd3"). InnerVolumeSpecName "kube-api-access-rxkh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:45:18.205240 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.205221 5755 scope.go:122] "RemoveContainer" containerID="47c9a016b5c6ab68999f3d255904a52becc9cdfaadad4c5097d4fe215eb3c4da" Oct 30 20:45:18.297676 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.297639 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5895ebd0-b52d-4de7-a0d9-699516c25dd3-maas-api-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:45:18.297676 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.297675 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxkh4\" (UniqueName: \"kubernetes.io/projected/5895ebd0-b52d-4de7-a0d9-699516c25dd3-kube-api-access-rxkh4\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:45:18.432277 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.432249 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:18.444660 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.444636 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:18.921226 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.921148 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" Oct 30 20:45:18.921226 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.921199 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7" event={"ID":"5895ebd0-b52d-4de7-a0d9-699516c25dd3","Type":"ContainerDied","Data":"9c2f7bcbfe573fb125104f2319236ad09d52dfa74738cc7694919ecce80254d6"} Oct 30 20:45:18.921765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.921265 5755 scope.go:122] "RemoveContainer" containerID="d22ea9f8457cee0e0891f4c082547555d00fc8506f34cf07adf7cd5424982144" Oct 30 20:45:18.923654 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.923623 5755 generic.go:356] "Generic (PLEG): container finished" podID="d25dea36-d864-42c7-a8e0-bc178bec1006" containerID="791c7bd442b1f4a850396f13524db310400f2397710e1649b489cc71f15b1cd8" exitCode=60 Oct 30 20:45:18.923775 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.923681 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" event={"ID":"d25dea36-d864-42c7-a8e0-bc178bec1006","Type":"ContainerDied","Data":"791c7bd442b1f4a850396f13524db310400f2397710e1649b489cc71f15b1cd8"} Oct 30 20:45:18.924093 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.924068 5755 scope.go:122] "RemoveContainer" containerID="791c7bd442b1f4a850396f13524db310400f2397710e1649b489cc71f15b1cd8" Oct 30 20:45:18.924355 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:45:18.924337 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6_odh-ai-gateway-infra(d25dea36-d864-42c7-a8e0-bc178bec1006)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" podUID="d25dea36-d864-42c7-a8e0-bc178bec1006" Oct 30 20:45:18.931059 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.931033 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" event={"ID":"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb","Type":"ContainerStarted","Data":"379a436ca4b90862ecdb5544742bf88c3d4deca3164ca3f5ea26d5fe81a688b9"} Oct 30 20:45:18.932626 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.932604 5755 scope.go:122] "RemoveContainer" containerID="07c24fa8c8ec6da3aa382a8ca21ef85444f43834098d7482ea37f9c7784a656d" Oct 30 20:45:18.944012 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:18.943985 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:45:19.069245 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.069218 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" Oct 30 20:45:19.159440 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.159403 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg"] Oct 30 20:45:19.159850 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.159838 5755 cpu_manager.go:414] "containerMap: removing container" podUID="5895ebd0-b52d-4de7-a0d9-699516c25dd3" containerName="cleanup" Oct 30 20:45:19.159897 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.159852 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5895ebd0-b52d-4de7-a0d9-699516c25dd3" containerName="cleanup" Oct 30 20:45:19.159897 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.159874 5755 cpu_manager.go:414] "containerMap: removing container" podUID="5895ebd0-b52d-4de7-a0d9-699516c25dd3" containerName="cleanup" Oct 30 20:45:19.159897 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.159878 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5895ebd0-b52d-4de7-a0d9-699516c25dd3" containerName="cleanup" Oct 30 20:45:19.159897 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.159889 5755 cpu_manager.go:414] "containerMap: removing container" podUID="5895ebd0-b52d-4de7-a0d9-699516c25dd3" containerName="cleanup" Oct 30 20:45:19.159897 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.159895 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5895ebd0-b52d-4de7-a0d9-699516c25dd3" containerName="cleanup" Oct 30 20:45:19.160065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.159906 5755 cpu_manager.go:414] "containerMap: removing container" podUID="e53b00f5-d023-4ea3-a163-f588849085ac" containerName="cleanup" Oct 30 20:45:19.160065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.159911 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e53b00f5-d023-4ea3-a163-f588849085ac" containerName="cleanup" Oct 30 20:45:19.160065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.159920 5755 cpu_manager.go:414] "containerMap: removing container" podUID="e53b00f5-d023-4ea3-a163-f588849085ac" containerName="cleanup" Oct 30 20:45:19.160065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.159925 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e53b00f5-d023-4ea3-a163-f588849085ac" containerName="cleanup" Oct 30 20:45:19.160065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.159932 5755 cpu_manager.go:414] "containerMap: removing container" podUID="e53b00f5-d023-4ea3-a163-f588849085ac" containerName="cleanup" Oct 30 20:45:19.160065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.159936 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e53b00f5-d023-4ea3-a163-f588849085ac" containerName="cleanup" Oct 30 20:45:19.160065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.160006 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="5895ebd0-b52d-4de7-a0d9-699516c25dd3" containerName="cleanup" Oct 30 20:45:19.160065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.160014 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="e53b00f5-d023-4ea3-a163-f588849085ac" containerName="cleanup" Oct 30 20:45:19.160065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.160020 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="e53b00f5-d023-4ea3-a163-f588849085ac" containerName="cleanup" Oct 30 20:45:19.160065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.160028 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="e53b00f5-d023-4ea3-a163-f588849085ac" containerName="cleanup" Oct 30 20:45:19.160065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.160036 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="5895ebd0-b52d-4de7-a0d9-699516c25dd3" containerName="cleanup" Oct 30 20:45:19.163225 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.163211 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.207825 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.207748 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e53b00f5-d023-4ea3-a163-f588849085ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53b00f5-d023-4ea3-a163-f588849085ac-maas-api-service-ca\") pod \"e53b00f5-d023-4ea3-a163-f588849085ac\" (UID: \"e53b00f5-d023-4ea3-a163-f588849085ac\") " Oct 30 20:45:19.207966 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.207877 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e53b00f5-d023-4ea3-a163-f588849085ac-kube-api-access-t24rm\" (UniqueName: \"kubernetes.io/projected/e53b00f5-d023-4ea3-a163-f588849085ac-kube-api-access-t24rm\") pod \"e53b00f5-d023-4ea3-a163-f588849085ac\" (UID: \"e53b00f5-d023-4ea3-a163-f588849085ac\") " Oct 30 20:45:19.208173 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.208146 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53b00f5-d023-4ea3-a163-f588849085ac-maas-api-service-ca" pod "e53b00f5-d023-4ea3-a163-f588849085ac" (UID: "e53b00f5-d023-4ea3-a163-f588849085ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:45:19.210225 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.210198 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53b00f5-d023-4ea3-a163-f588849085ac-kube-api-access-t24rm" pod "e53b00f5-d023-4ea3-a163-f588849085ac" (UID: "e53b00f5-d023-4ea3-a163-f588849085ac"). InnerVolumeSpecName "kube-api-access-t24rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:45:19.224622 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.224600 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b43c84\"/\"kube-root-ca.crt\"" Oct 30 20:45:19.224832 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.224809 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b43c84\"/\"default-dockercfg-jgd8f\"" Oct 30 20:45:19.224923 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.224820 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b43c84\"/\"openshift-service-ca.crt\"" Oct 30 20:45:19.224923 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.224909 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b43c84\"/\"distinct-2-w2-b43c84-kserve-self-signed-certs\"" Oct 30 20:45:19.228314 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.228291 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7"] Oct 30 20:45:19.277211 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.277175 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg"] Oct 30 20:45:19.279479 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.279456 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b43c84-29889885-jc5n7"] Oct 30 20:45:19.281435 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.281416 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6"] Oct 30 20:45:19.309234 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.309200 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9twdk\" (UniqueName: \"kubernetes.io/projected/51e57bd5-935c-4aac-b511-bbfd0068ebb8-kube-api-access-9twdk\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.309431 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.309316 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-dshm\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.309431 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.309368 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-model-cache\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.309536 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.309476 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e57bd5-935c-4aac-b511-bbfd0068ebb8-tls-certs\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.309578 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.309536 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-tmp-dir\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.309626 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.309577 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-home\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.309742 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.309730 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t24rm\" (UniqueName: \"kubernetes.io/projected/e53b00f5-d023-4ea3-a163-f588849085ac-kube-api-access-t24rm\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:45:19.309791 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.309746 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53b00f5-d023-4ea3-a163-f588849085ac-maas-api-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:45:19.411048 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.411013 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9twdk\" (UniqueName: \"kubernetes.io/projected/51e57bd5-935c-4aac-b511-bbfd0068ebb8-kube-api-access-9twdk\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.411268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.411111 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-dshm\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.411268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.411177 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-model-cache\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.411268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.411217 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e57bd5-935c-4aac-b511-bbfd0068ebb8-tls-certs\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.411268 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.411259 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-tmp-dir\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.411630 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.411291 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-home\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.412111 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.412072 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-home\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.413147 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.413121 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-tmp-dir\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.413425 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.413405 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-model-cache\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.414322 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.414276 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-dshm\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.414914 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.414886 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e57bd5-935c-4aac-b511-bbfd0068ebb8-tls-certs\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.471105 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.471071 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9twdk\" (UniqueName: \"kubernetes.io/projected/51e57bd5-935c-4aac-b511-bbfd0068ebb8-kube-api-access-9twdk\") pod \"distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.474932 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.474912 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:19.943000 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.942960 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" event={"ID":"e53b00f5-d023-4ea3-a163-f588849085ac","Type":"ContainerDied","Data":"6c926312390ac79c787475bfa4c2ce97836023124407956059ce3e79d2c7f73f"} Oct 30 20:45:19.944065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.943080 5755 scope.go:122] "RemoveContainer" containerID="222bd21dc673ff8588909b61ca959c55b73cc1f120150674383e886ee47ad7a7" Oct 30 20:45:19.944065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.942981 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4" Oct 30 20:45:19.945672 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.945638 5755 generic.go:356] "Generic (PLEG): container finished" podID="2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" containerID="379a436ca4b90862ecdb5544742bf88c3d4deca3164ca3f5ea26d5fe81a688b9" exitCode=60 Oct 30 20:45:19.945834 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.945770 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" event={"ID":"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb","Type":"ContainerDied","Data":"379a436ca4b90862ecdb5544742bf88c3d4deca3164ca3f5ea26d5fe81a688b9"} Oct 30 20:45:19.946234 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.946219 5755 scope.go:122] "RemoveContainer" containerID="379a436ca4b90862ecdb5544742bf88c3d4deca3164ca3f5ea26d5fe81a688b9" Oct 30 20:45:19.946449 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:45:19.946431 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv_odh-ai-gateway-infra(2d6bc14f-d81b-4b3e-80fb-28be2cef44bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" podUID="2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" Oct 30 20:45:19.955717 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.955695 5755 scope.go:122] "RemoveContainer" containerID="47c9a016b5c6ab68999f3d255904a52becc9cdfaadad4c5097d4fe215eb3c4da" Oct 30 20:45:19.988583 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:19.988556 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg"] Oct 30 20:45:19.989582 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:45:19.989553 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51e57bd5_935c_4aac_b511_bbfd0068ebb8.slice/crio-16298d11d6dbf3e95dfcaf7d399712f4447d659ccc0700a6dbc0a1be11f2775e WatchSource:0}: Error finding container 16298d11d6dbf3e95dfcaf7d399712f4447d659ccc0700a6dbc0a1be11f2775e: Status 404 returned error can't find the container with id 16298d11d6dbf3e95dfcaf7d399712f4447d659ccc0700a6dbc0a1be11f2775e Oct 30 20:45:20.100871 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:20.100835 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" Oct 30 20:45:20.221468 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:20.221434 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25dea36-d864-42c7-a8e0-bc178bec1006-kube-api-access-ckffs\" (UniqueName: \"kubernetes.io/projected/d25dea36-d864-42c7-a8e0-bc178bec1006-kube-api-access-ckffs\") pod \"d25dea36-d864-42c7-a8e0-bc178bec1006\" (UID: \"d25dea36-d864-42c7-a8e0-bc178bec1006\") " Oct 30 20:45:20.221614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:20.221495 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d25dea36-d864-42c7-a8e0-bc178bec1006-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25dea36-d864-42c7-a8e0-bc178bec1006-maas-api-service-ca\") pod \"d25dea36-d864-42c7-a8e0-bc178bec1006\" (UID: \"d25dea36-d864-42c7-a8e0-bc178bec1006\") " Oct 30 20:45:20.221904 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:20.221873 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25dea36-d864-42c7-a8e0-bc178bec1006-maas-api-service-ca" pod "d25dea36-d864-42c7-a8e0-bc178bec1006" (UID: "d25dea36-d864-42c7-a8e0-bc178bec1006"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:45:20.223692 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:20.223670 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25dea36-d864-42c7-a8e0-bc178bec1006-kube-api-access-ckffs" pod "d25dea36-d864-42c7-a8e0-bc178bec1006" (UID: "d25dea36-d864-42c7-a8e0-bc178bec1006"). InnerVolumeSpecName "kube-api-access-ckffs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:45:20.283232 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:20.283196 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4"] Oct 30 20:45:20.288785 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:20.288756 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-395022-29889885-kvbs4"] Oct 30 20:45:20.323128 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:20.323093 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckffs\" (UniqueName: \"kubernetes.io/projected/d25dea36-d864-42c7-a8e0-bc178bec1006-kube-api-access-ckffs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:45:20.323128 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:20.323129 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25dea36-d864-42c7-a8e0-bc178bec1006-maas-api-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:45:20.517405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:20.517307 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv"] Oct 30 20:45:20.953124 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:20.953038 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" event={"ID":"51e57bd5-935c-4aac-b511-bbfd0068ebb8","Type":"ContainerStarted","Data":"1a1e777669ec876b4b0e944b38b34bb5c4e04994185e100d768974505fb12b16"} Oct 30 20:45:20.953124 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:20.953084 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" event={"ID":"51e57bd5-935c-4aac-b511-bbfd0068ebb8","Type":"ContainerStarted","Data":"16298d11d6dbf3e95dfcaf7d399712f4447d659ccc0700a6dbc0a1be11f2775e"} Oct 30 20:45:20.954450 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:20.954409 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" event={"ID":"d25dea36-d864-42c7-a8e0-bc178bec1006","Type":"ContainerDied","Data":"804e173b9078d6068de81b262832f53c90e7abbcbc03e821674b3504ba7c28a3"} Oct 30 20:45:20.954594 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:20.954420 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6" Oct 30 20:45:20.954594 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:20.954484 5755 scope.go:122] "RemoveContainer" containerID="791c7bd442b1f4a850396f13524db310400f2397710e1649b489cc71f15b1cd8" Oct 30 20:45:21.070824 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:21.070777 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podStartSLOduration=2.070763449 podStartE2EDuration="2.070763449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:21.069248292 +0000 UTC m=+1083.995805207" watchObservedRunningTime="2026-10-30 20:45:21.070763449 +0000 UTC m=+1083.997320363" Oct 30 20:45:21.102527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:21.102501 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" Oct 30 20:45:21.212750 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:21.212678 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5895ebd0-b52d-4de7-a0d9-699516c25dd3" path="/var/lib/kubelet/pods/5895ebd0-b52d-4de7-a0d9-699516c25dd3/volumes" Oct 30 20:45:21.213057 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:21.213044 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e53b00f5-d023-4ea3-a163-f588849085ac" path="/var/lib/kubelet/pods/e53b00f5-d023-4ea3-a163-f588849085ac/volumes" Oct 30 20:45:21.234037 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:21.234008 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d6bc14f-d81b-4b3e-80fb-28be2cef44bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6bc14f-d81b-4b3e-80fb-28be2cef44bb-maas-api-service-ca\") pod \"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb\" (UID: \"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb\") " Oct 30 20:45:21.234162 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:21.234095 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6bc14f-d81b-4b3e-80fb-28be2cef44bb-kube-api-access-cr8jg\" (UniqueName: \"kubernetes.io/projected/2d6bc14f-d81b-4b3e-80fb-28be2cef44bb-kube-api-access-cr8jg\") pod \"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb\" (UID: \"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb\") " Oct 30 20:45:21.234434 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:21.234410 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6bc14f-d81b-4b3e-80fb-28be2cef44bb-maas-api-service-ca" pod "2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" (UID: "2d6bc14f-d81b-4b3e-80fb-28be2cef44bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:45:21.236361 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:21.236340 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6bc14f-d81b-4b3e-80fb-28be2cef44bb-kube-api-access-cr8jg" pod "2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" (UID: "2d6bc14f-d81b-4b3e-80fb-28be2cef44bb"). InnerVolumeSpecName "kube-api-access-cr8jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:45:21.299372 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:21.299339 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6"] Oct 30 20:45:21.307149 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:21.307126 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b32941-29889885-wsww6"] Oct 30 20:45:21.336130 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:21.336106 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6bc14f-d81b-4b3e-80fb-28be2cef44bb-maas-api-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:45:21.336130 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:21.336127 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr8jg\" (UniqueName: \"kubernetes.io/projected/2d6bc14f-d81b-4b3e-80fb-28be2cef44bb-kube-api-access-cr8jg\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:45:21.475281 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:21.475246 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:21.966464 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:21.966366 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" Oct 30 20:45:21.966464 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:21.966414 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv" event={"ID":"2d6bc14f-d81b-4b3e-80fb-28be2cef44bb","Type":"ContainerDied","Data":"5405eacb74d3d6777a4b209089ed05db944d9aa8b67efe0dc52643790c766eee"} Oct 30 20:45:21.966956 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:21.966472 5755 scope.go:122] "RemoveContainer" containerID="379a436ca4b90862ecdb5544742bf88c3d4deca3164ca3f5ea26d5fe81a688b9" Oct 30 20:45:22.140467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:22.140423 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv"] Oct 30 20:45:22.140673 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:22.140481 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-86d620-29889885-lqbsv"] Oct 30 20:45:23.212292 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:23.212261 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" path="/var/lib/kubelet/pods/2d6bc14f-d81b-4b3e-80fb-28be2cef44bb/volumes" Oct 30 20:45:23.212715 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:23.212700 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d25dea36-d864-42c7-a8e0-bc178bec1006" path="/var/lib/kubelet/pods/d25dea36-d864-42c7-a8e0-bc178bec1006/volumes" Oct 30 20:45:29.475341 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:29.475261 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:29.487782 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:29.487752 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:30.025154 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:30.025123 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:45:32.965004 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.964967 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9"] Oct 30 20:45:32.965494 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.965380 5755 cpu_manager.go:414] "containerMap: removing container" podUID="d25dea36-d864-42c7-a8e0-bc178bec1006" containerName="cleanup" Oct 30 20:45:32.965494 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.965406 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25dea36-d864-42c7-a8e0-bc178bec1006" containerName="cleanup" Oct 30 20:45:32.965494 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.965419 5755 cpu_manager.go:414] "containerMap: removing container" podUID="2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" containerName="cleanup" Oct 30 20:45:32.965494 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.965423 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" containerName="cleanup" Oct 30 20:45:32.965494 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.965437 5755 cpu_manager.go:414] "containerMap: removing container" podUID="d25dea36-d864-42c7-a8e0-bc178bec1006" containerName="cleanup" Oct 30 20:45:32.965494 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.965441 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25dea36-d864-42c7-a8e0-bc178bec1006" containerName="cleanup" Oct 30 20:45:32.965494 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.965468 5755 cpu_manager.go:414] "containerMap: removing container" podUID="d25dea36-d864-42c7-a8e0-bc178bec1006" containerName="cleanup" Oct 30 20:45:32.965494 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.965473 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25dea36-d864-42c7-a8e0-bc178bec1006" containerName="cleanup" Oct 30 20:45:32.965494 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.965491 5755 cpu_manager.go:414] "containerMap: removing container" podUID="2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" containerName="cleanup" Oct 30 20:45:32.965494 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.965496 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" containerName="cleanup" Oct 30 20:45:32.966038 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.965577 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25dea36-d864-42c7-a8e0-bc178bec1006" containerName="cleanup" Oct 30 20:45:32.966038 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.965585 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="5895ebd0-b52d-4de7-a0d9-699516c25dd3" containerName="cleanup" Oct 30 20:45:32.966038 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.965590 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" containerName="cleanup" Oct 30 20:45:32.966038 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.965596 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" containerName="cleanup" Oct 30 20:45:32.966038 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.965604 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25dea36-d864-42c7-a8e0-bc178bec1006" containerName="cleanup" Oct 30 20:45:32.968711 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:32.968691 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.063070 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.063036 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-home\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.063218 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.063162 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-dshm\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.063218 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.063196 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-model-cache\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.063312 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.063272 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t82bw\" (UniqueName: \"kubernetes.io/projected/440b1853-8437-4777-9b63-afddbbb3f89a-kube-api-access-t82bw\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.063312 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.063307 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-tmp-dir\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.063375 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.063322 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/440b1853-8437-4777-9b63-afddbbb3f89a-tls-certs\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.073833 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.073817 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ee3fad\"/\"distinct-2-w6-ee3fad-kserve-self-signed-certs\"" Oct 30 20:45:33.076413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.076371 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9"] Oct 30 20:45:33.164024 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.163987 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-dshm\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.164024 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.164028 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-model-cache\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.164278 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.164050 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t82bw\" (UniqueName: \"kubernetes.io/projected/440b1853-8437-4777-9b63-afddbbb3f89a-kube-api-access-t82bw\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.164278 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.164072 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-tmp-dir\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.164278 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.164089 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/440b1853-8437-4777-9b63-afddbbb3f89a-tls-certs\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.164278 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.164196 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-home\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.164534 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.164509 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-model-cache\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.164590 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.164557 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-tmp-dir\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.164638 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.164624 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-home\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.166612 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.166583 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-dshm\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.166820 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.166802 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/440b1853-8437-4777-9b63-afddbbb3f89a-tls-certs\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.222840 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.222809 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t82bw\" (UniqueName: \"kubernetes.io/projected/440b1853-8437-4777-9b63-afddbbb3f89a-kube-api-access-t82bw\") pod \"distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.280906 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.280872 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:33.679013 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:33.678978 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9"] Oct 30 20:45:33.680966 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:45:33.680940 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod440b1853_8437_4777_9b63_afddbbb3f89a.slice/crio-b48e5a8901f0eb10da1feccb8cf9b2624929468dfb1ba06b5d8b5140b310c981 WatchSource:0}: Error finding container b48e5a8901f0eb10da1feccb8cf9b2624929468dfb1ba06b5d8b5140b310c981: Status 404 returned error can't find the container with id b48e5a8901f0eb10da1feccb8cf9b2624929468dfb1ba06b5d8b5140b310c981 Oct 30 20:45:34.041471 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:34.041421 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" event={"ID":"440b1853-8437-4777-9b63-afddbbb3f89a","Type":"ContainerStarted","Data":"9277f31e4a8bf2717b0350b4c0eff63deadbc52afb5c345df700bbf6c7571fb6"} Oct 30 20:45:34.041885 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:34.041480 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" event={"ID":"440b1853-8437-4777-9b63-afddbbb3f89a","Type":"ContainerStarted","Data":"b48e5a8901f0eb10da1feccb8cf9b2624929468dfb1ba06b5d8b5140b310c981"} Oct 30 20:45:34.158776 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:34.158714 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podStartSLOduration=2.158669476 podStartE2EDuration="2.158669476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:34.156298494 +0000 UTC m=+1097.082855431" watchObservedRunningTime="2026-10-30 20:45:34.158669476 +0000 UTC m=+1097.085226395" Oct 30 20:45:34.281039 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:34.281006 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:41.255956 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:45:41.255907 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:45:43.281976 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:43.281939 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:43.295192 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:43.295170 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:44.110187 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:44.110158 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:45:51.067789 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.067749 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh"] Oct 30 20:45:51.068262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.068207 5755 cpu_manager.go:414] "containerMap: removing container" podUID="2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" containerName="cleanup" Oct 30 20:45:51.068262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.068219 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" containerName="cleanup" Oct 30 20:45:51.068333 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.068327 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25dea36-d864-42c7-a8e0-bc178bec1006" containerName="cleanup" Oct 30 20:45:51.068368 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.068340 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6bc14f-d81b-4b3e-80fb-28be2cef44bb" containerName="cleanup" Oct 30 20:45:51.072848 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.072829 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.167439 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.167377 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mdn7\" (UniqueName: \"kubernetes.io/projected/0be44dfb-bf6d-48a8-b061-71899ad1587f-kube-api-access-5mdn7\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.167439 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.167444 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-dshm\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.167695 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.167536 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-tmp-dir\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.167695 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.167597 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-home\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.167695 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.167682 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0be44dfb-bf6d-48a8-b061-71899ad1587f-tls-certs\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.167856 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.167790 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-model-cache\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.177101 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.177071 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-395022\"/\"kube-root-ca.crt\"" Oct 30 20:45:51.177101 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.177081 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-395022\"/\"default-dockercfg-gqktn\"" Oct 30 20:45:51.177279 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.177135 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-395022\"/\"distinct-2-w4-395022-kserve-self-signed-certs\"" Oct 30 20:45:51.177653 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.177638 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-395022\"/\"openshift-service-ca.crt\"" Oct 30 20:45:51.180442 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.180418 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh"] Oct 30 20:45:51.269047 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.269016 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mdn7\" (UniqueName: \"kubernetes.io/projected/0be44dfb-bf6d-48a8-b061-71899ad1587f-kube-api-access-5mdn7\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.269258 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.269055 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-dshm\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.269258 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.269087 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-tmp-dir\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.269258 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.269113 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-home\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.269258 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.269169 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0be44dfb-bf6d-48a8-b061-71899ad1587f-tls-certs\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.269538 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.269261 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-model-cache\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.269676 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.269651 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-tmp-dir\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.269807 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.269692 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-home\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.269807 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.269786 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-model-cache\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.271574 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.271551 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-dshm\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.271877 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.271857 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0be44dfb-bf6d-48a8-b061-71899ad1587f-tls-certs\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.333236 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.333161 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mdn7\" (UniqueName: \"kubernetes.io/projected/0be44dfb-bf6d-48a8-b061-71899ad1587f-kube-api-access-5mdn7\") pod \"distinct-2-w4-395022-kserve-786f767c59-pvskh\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.385759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.385731 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:45:51.770214 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:51.770191 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh"] Oct 30 20:45:51.771773 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:45:51.771751 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0be44dfb_bf6d_48a8_b061_71899ad1587f.slice/crio-fce01604081d808af25751fb3c3a4e1ce5cf25ac74f87ee3fd4f6051415fcc22 WatchSource:0}: Error finding container fce01604081d808af25751fb3c3a4e1ce5cf25ac74f87ee3fd4f6051415fcc22: Status 404 returned error can't find the container with id fce01604081d808af25751fb3c3a4e1ce5cf25ac74f87ee3fd4f6051415fcc22 Oct 30 20:45:52.150692 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:52.150605 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" event={"ID":"0be44dfb-bf6d-48a8-b061-71899ad1587f","Type":"ContainerStarted","Data":"c41d2c3effc96de25720ea854ac7db321953e37b4c9cc9daea516230d82699a1"} Oct 30 20:45:52.150692 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:52.150645 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" event={"ID":"0be44dfb-bf6d-48a8-b061-71899ad1587f","Type":"ContainerStarted","Data":"fce01604081d808af25751fb3c3a4e1ce5cf25ac74f87ee3fd4f6051415fcc22"} Oct 30 20:45:52.268690 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:52.268638 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podStartSLOduration=1.268624155 podStartE2EDuration="1.268624155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:52.26655669 +0000 UTC m=+1115.193113663" watchObservedRunningTime="2026-10-30 20:45:52.268624155 +0000 UTC m=+1115.195181091" Oct 30 20:45:52.386326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:45:52.386001 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:46:01.386002 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:01.385968 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:46:01.398436 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:01.398405 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:46:02.230053 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:02.230024 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:46:11.255638 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:46:11.255520 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:46:14.427588 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.427558 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr"] Oct 30 20:46:14.431269 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.431249 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.484931 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.484899 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-86d620-openshift-default-dockercfg-b92ff\"" Oct 30 20:46:14.522853 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.522811 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.523023 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.522867 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.523023 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.522997 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.523100 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.523039 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.523100 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.523086 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c12235-feb4-43a4-a2e3-32eb61584f28-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.523190 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.523175 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.523246 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.523228 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.523297 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.523262 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.523414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.523373 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl4l5\" (UniqueName: \"kubernetes.io/projected/56c12235-feb4-43a4-a2e3-32eb61584f28-kube-api-access-zl4l5\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.537333 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.537305 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr"] Oct 30 20:46:14.624503 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.624470 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.624705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.624513 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.624705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.624546 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c12235-feb4-43a4-a2e3-32eb61584f28-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.624705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.624599 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.624705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.624627 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.624705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.624653 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.624705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.624690 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl4l5\" (UniqueName: \"kubernetes.io/projected/56c12235-feb4-43a4-a2e3-32eb61584f28-kube-api-access-zl4l5\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.625071 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.624772 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.625071 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.624817 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.625071 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.624998 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.625237 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.625174 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.625237 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.625221 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.625442 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.625420 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c12235-feb4-43a4-a2e3-32eb61584f28-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.625569 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.625452 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.626932 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.626911 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.627314 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.627296 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.685071 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.684971 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd"] Oct 30 20:46:14.689874 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.689838 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl4l5\" (UniqueName: \"kubernetes.io/projected/56c12235-feb4-43a4-a2e3-32eb61584f28-kube-api-access-zl4l5\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.692633 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.692600 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.695851 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.695823 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.745318 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.745288 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:14.752226 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.752196 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd"] Oct 30 20:46:14.752619 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.752594 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ee3fad-openshift-default-dockercfg-tzrhl\"" Oct 30 20:46:14.828085 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.828048 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.828249 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.828124 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.828249 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.828168 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.828249 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.828201 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.828400 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.828297 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cknwv\" (UniqueName: \"kubernetes.io/projected/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-kube-api-access-cknwv\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.828400 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.828367 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.828481 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.828464 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.828529 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.828490 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.828564 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.828530 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.929273 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.929239 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.929273 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.929280 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.929518 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.929301 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.929518 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.929378 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cknwv\" (UniqueName: \"kubernetes.io/projected/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-kube-api-access-cknwv\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.929518 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.929480 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.929645 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.929544 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.929645 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.929568 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.929645 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.929604 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.929791 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.929646 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.929791 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.929741 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.929791 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.929784 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.929943 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.929822 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.930053 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.930030 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.930303 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.930284 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.931844 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.931823 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.931980 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.931964 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.990343 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.990298 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:14.990523 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:14.990474 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cknwv\" (UniqueName: \"kubernetes.io/projected/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-kube-api-access-cknwv\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:15.022595 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:15.022565 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:15.189096 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:46:15.189068 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c12235_feb4_43a4_a2e3_32eb61584f28.slice/crio-8094d6dd692770ac61c3a6308062a907cd6d5f136e218555e506855740302e7c WatchSource:0}: Error finding container 8094d6dd692770ac61c3a6308062a907cd6d5f136e218555e506855740302e7c: Status 404 returned error can't find the container with id 8094d6dd692770ac61c3a6308062a907cd6d5f136e218555e506855740302e7c Oct 30 20:46:15.189096 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:15.189085 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr"] Oct 30 20:46:15.191366 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:15.191325 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:15.191473 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:15.191422 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:15.191473 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:15.191452 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:15.300716 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:15.300676 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" event={"ID":"56c12235-feb4-43a4-a2e3-32eb61584f28","Type":"ContainerStarted","Data":"2b7ce88e34380ab18cb6b287ea469dcd2d98050831c840d895233b501c8257e8"} Oct 30 20:46:15.300917 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:15.300740 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" event={"ID":"56c12235-feb4-43a4-a2e3-32eb61584f28","Type":"ContainerStarted","Data":"8094d6dd692770ac61c3a6308062a907cd6d5f136e218555e506855740302e7c"} Oct 30 20:46:15.406080 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:46:15.406049 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffff493d_18a3_4ca2_adc9_3bdc3b51c30a.slice/crio-221aeb4591a6acb910e4290d75e6472af4f67c690b17ea95ebd7726184061a98 WatchSource:0}: Error finding container 221aeb4591a6acb910e4290d75e6472af4f67c690b17ea95ebd7726184061a98: Status 404 returned error can't find the container with id 221aeb4591a6acb910e4290d75e6472af4f67c690b17ea95ebd7726184061a98 Oct 30 20:46:15.406350 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:15.406322 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd"] Oct 30 20:46:15.408772 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:15.408737 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:15.408884 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:15.408818 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:15.408884 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:15.408863 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:15.422214 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:15.422149 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" podStartSLOduration=1.4221331560000001 podStartE2EDuration="1.422133156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:15.420968679 +0000 UTC m=+1138.347525620" watchObservedRunningTime="2026-10-30 20:46:15.422133156 +0000 UTC m=+1138.348690073" Oct 30 20:46:15.745771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:15.745739 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:16.326917 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.326882 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" event={"ID":"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a","Type":"ContainerStarted","Data":"fbcf848300a612089d7ab94cc57a0c245c5e897c97455a9111626c4c78eebef6"} Oct 30 20:46:16.327183 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.327163 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" event={"ID":"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a","Type":"ContainerStarted","Data":"221aeb4591a6acb910e4290d75e6472af4f67c690b17ea95ebd7726184061a98"} Oct 30 20:46:16.464966 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.464905 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" podStartSLOduration=2.4648867709999998 podStartE2EDuration="2.464886771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:16.462798837 +0000 UTC m=+1139.389355753" watchObservedRunningTime="2026-10-30 20:46:16.464886771 +0000 UTC m=+1139.391443687" Oct 30 20:46:16.605479 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.604874 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22"] Oct 30 20:46:16.627430 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.616677 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.654199 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.654170 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.654355 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.654223 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.654420 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.654355 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48jhc\" (UniqueName: \"kubernetes.io/projected/00f11563-7c2c-4c14-a6a9-1b2f751a057d-kube-api-access-48jhc\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.654498 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.654480 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.654534 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.654524 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.654589 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.654567 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.654825 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.654801 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.654896 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.654837 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.654896 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.654877 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.732979 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.729249 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22"] Oct 30 20:46:16.747614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.747580 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:46:16.748009 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.747635 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" podUID="56c12235-feb4-43a4-a2e3-32eb61584f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:46:16.755837 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.755810 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.755972 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.755849 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.755972 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.755888 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.755972 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.755953 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.756104 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.756007 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.756104 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.756057 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48jhc\" (UniqueName: \"kubernetes.io/projected/00f11563-7c2c-4c14-a6a9-1b2f751a057d-kube-api-access-48jhc\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.756161 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.756110 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.756161 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.756146 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.756227 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.756179 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.756987 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.756966 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.758212 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.758186 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.758703 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.758680 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.758822 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.758723 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.758883 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.758858 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.759020 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.759004 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.760552 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.760527 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.827299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.827273 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48jhc\" (UniqueName: \"kubernetes.io/projected/00f11563-7c2c-4c14-a6a9-1b2f751a057d-kube-api-access-48jhc\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.829076 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.829048 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:16.939788 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:16.939704 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:17.023052 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:17.023007 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:17.024191 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:17.024120 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:46:17.024191 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:17.024163 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" podUID="ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 20:46:17.427438 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:17.427408 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22"] Oct 30 20:46:17.436138 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:17.435561 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:17.436138 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:17.435646 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:17.436138 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:17.435685 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:17.747687 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:17.747650 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:46:17.748194 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:17.747720 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" podUID="56c12235-feb4-43a4-a2e3-32eb61584f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:46:18.025477 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:18.025234 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:46:18.025477 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:18.025293 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" podUID="ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 20:46:18.359244 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:18.358796 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" event={"ID":"00f11563-7c2c-4c14-a6a9-1b2f751a057d","Type":"ContainerStarted","Data":"4c56332dd37ad287e96900b62b4e08f8455a1ea218cd9516cbbe4ff472659aeb"} Oct 30 20:46:18.359244 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:18.358851 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" event={"ID":"00f11563-7c2c-4c14-a6a9-1b2f751a057d","Type":"ContainerStarted","Data":"0e836cd62067a2e868404d8332ff863a991e9754dac164958d12fb161fee8efb"} Oct 30 20:46:18.486412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:18.483729 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" podStartSLOduration=2.483710244 podStartE2EDuration="2.483710244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:18.482718452 +0000 UTC m=+1141.409275396" watchObservedRunningTime="2026-10-30 20:46:18.483710244 +0000 UTC m=+1141.410267149" Oct 30 20:46:18.748895 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:18.748837 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:46:18.749303 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:18.748903 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" podUID="56c12235-feb4-43a4-a2e3-32eb61584f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:46:18.947415 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:18.946018 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:18.947415 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:18.946146 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:46:18.947415 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:18.946189 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" podUID="00f11563-7c2c-4c14-a6a9-1b2f751a057d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 20:46:19.044471 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:19.036654 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:46:19.044471 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:19.036713 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" podUID="ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 20:46:19.747407 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:19.746937 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:46:19.747407 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:19.746998 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" podUID="56c12235-feb4-43a4-a2e3-32eb61584f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:46:19.943795 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:19.943757 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:46:19.944378 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:19.944343 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" podUID="00f11563-7c2c-4c14-a6a9-1b2f751a057d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 20:46:20.023931 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:20.023834 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:46:20.024239 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:20.024208 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" podUID="ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 20:46:20.748014 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:20.745716 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:46:20.748014 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:20.745778 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" podUID="56c12235-feb4-43a4-a2e3-32eb61584f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:46:20.946413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:20.942838 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:46:20.946413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:20.942903 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" podUID="00f11563-7c2c-4c14-a6a9-1b2f751a057d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 20:46:21.037480 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:21.030843 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:46:21.037480 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:21.030911 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" podUID="ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 20:46:21.746053 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:21.746018 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:46:21.746306 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:21.746070 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" podUID="56c12235-feb4-43a4-a2e3-32eb61584f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:46:21.940718 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:21.940683 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:46:21.940929 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:21.940736 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" podUID="00f11563-7c2c-4c14-a6a9-1b2f751a057d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 20:46:22.023810 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:22.023718 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:46:22.023810 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:22.023788 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" podUID="ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 20:46:22.898292 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:22.896697 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:22.898292 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:22.898139 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:22.906819 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:22.906705 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:46:22.940489 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:22.940452 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:46:22.940678 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:22.940527 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" podUID="00f11563-7c2c-4c14-a6a9-1b2f751a057d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 20:46:23.162752 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:23.162655 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:23.163307 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:23.163274 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:23.164756 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:23.164734 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 20:46:23.941175 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:23.940763 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:46:23.941175 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:23.940823 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" podUID="00f11563-7c2c-4c14-a6a9-1b2f751a057d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 20:46:25.169492 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:25.169449 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:25.169969 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:25.169844 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:25.171924 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:25.171902 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:46:29.571808 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.571768 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96"] Oct 30 20:46:29.576156 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.576129 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.683698 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.683661 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96"] Oct 30 20:46:29.697430 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.697373 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42lmw\" (UniqueName: \"kubernetes.io/projected/e41e4f75-ea02-41f4-9698-c064cefa94ab-kube-api-access-42lmw\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.697626 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.697502 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41e4f75-ea02-41f4-9698-c064cefa94ab-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.697626 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.697543 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.697749 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.697662 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.697749 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.697728 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.697835 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.697755 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.697835 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.697785 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.697916 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.697868 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.697916 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.697896 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.799704 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.799663 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.799918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.799749 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.799918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.799772 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.799918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.799796 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.799918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.799857 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.800140 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.799952 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.800140 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.800098 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42lmw\" (UniqueName: \"kubernetes.io/projected/e41e4f75-ea02-41f4-9698-c064cefa94ab-kube-api-access-42lmw\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.800321 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.800138 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41e4f75-ea02-41f4-9698-c064cefa94ab-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.800321 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.800145 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.800321 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.800169 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.800321 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.800181 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.800321 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.800224 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.800857 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.800364 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.800857 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.800831 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41e4f75-ea02-41f4-9698-c064cefa94ab-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.802404 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.802367 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.802542 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.802517 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.866355 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.866266 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.869023 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.868991 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42lmw\" (UniqueName: \"kubernetes.io/projected/e41e4f75-ea02-41f4-9698-c064cefa94ab-kube-api-access-42lmw\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:29.890943 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:29.890908 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:30.329269 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:30.329240 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96"] Oct 30 20:46:30.331542 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:46:30.331506 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41e4f75_ea02_41f4_9698_c064cefa94ab.slice/crio-77aa3fcdd1fc77f28ec178b27f62e5f2e2ef1db1fcbc8e413f961ac9d4eba27a WatchSource:0}: Error finding container 77aa3fcdd1fc77f28ec178b27f62e5f2e2ef1db1fcbc8e413f961ac9d4eba27a: Status 404 returned error can't find the container with id 77aa3fcdd1fc77f28ec178b27f62e5f2e2ef1db1fcbc8e413f961ac9d4eba27a Oct 30 20:46:30.334440 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:30.334380 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:30.334527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:30.334510 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:30.334572 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:30.334560 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:30.466655 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:30.466620 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" event={"ID":"e41e4f75-ea02-41f4-9698-c064cefa94ab","Type":"ContainerStarted","Data":"7bfc15a85c4b8764a870f683caf0ab9da019838b40582cf81156762ec6fb0c81"} Oct 30 20:46:30.466876 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:30.466676 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" event={"ID":"e41e4f75-ea02-41f4-9698-c064cefa94ab","Type":"ContainerStarted","Data":"77aa3fcdd1fc77f28ec178b27f62e5f2e2ef1db1fcbc8e413f961ac9d4eba27a"} Oct 30 20:46:30.589317 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:30.589194 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" podStartSLOduration=1.58917238 podStartE2EDuration="1.58917238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:30.585690025 +0000 UTC m=+1153.512246973" watchObservedRunningTime="2026-10-30 20:46:30.58917238 +0000 UTC m=+1153.515729296" Oct 30 20:46:30.892284 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:30.892195 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:31.896583 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:31.896543 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:46:31.897189 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:31.897154 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" podUID="e41e4f75-ea02-41f4-9698-c064cefa94ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 20:46:32.912415 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:32.895860 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:46:32.912415 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:32.895935 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" podUID="e41e4f75-ea02-41f4-9698-c064cefa94ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 20:46:33.584529 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:33.584486 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:46:33.746408 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:33.745577 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:46:33.827567 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:33.826483 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:46:33.895519 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:33.895422 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:46:33.895780 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:33.895745 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" podUID="e41e4f75-ea02-41f4-9698-c064cefa94ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 20:46:34.894865 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:34.894830 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:46:34.895371 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:34.894890 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" podUID="e41e4f75-ea02-41f4-9698-c064cefa94ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 20:46:35.894412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:35.893829 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:46:35.894412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:35.893893 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" podUID="e41e4f75-ea02-41f4-9698-c064cefa94ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 20:46:36.892148 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:36.892106 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:46:36.892588 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:36.892174 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" podUID="e41e4f75-ea02-41f4-9698-c064cefa94ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 20:46:38.014374 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:38.014340 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:38.014862 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:38.014730 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:38.015704 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:38.015687 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:46:41.258359 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:46:41.258320 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:46:49.085657 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:49.085616 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf"] Oct 30 20:46:49.090806 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:49.090782 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf" Oct 30 20:46:49.146847 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:49.146812 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf"] Oct 30 20:46:49.174656 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:49.174629 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7fz4\" (UniqueName: \"kubernetes.io/projected/207b3254-36dd-450e-841a-bc771bf4ac3a-kube-api-access-m7fz4\") pod \"maas-api-revoke-sub-a8484f78-k2xzf\" (UID: \"207b3254-36dd-450e-841a-bc771bf4ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf" Oct 30 20:46:49.174815 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:49.174669 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/207b3254-36dd-450e-841a-bc771bf4ac3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8484f78-k2xzf\" (UID: \"207b3254-36dd-450e-841a-bc771bf4ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf" Oct 30 20:46:49.174815 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:49.174693 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/207b3254-36dd-450e-841a-bc771bf4ac3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8484f78-k2xzf\" (UID: \"207b3254-36dd-450e-841a-bc771bf4ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf" Oct 30 20:46:49.193006 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:49.192983 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:46:49.193006 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:49.192989 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:46:49.276722 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:49.276693 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fz4\" (UniqueName: \"kubernetes.io/projected/207b3254-36dd-450e-841a-bc771bf4ac3a-kube-api-access-m7fz4\") pod \"maas-api-revoke-sub-a8484f78-k2xzf\" (UID: \"207b3254-36dd-450e-841a-bc771bf4ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf" Oct 30 20:46:49.276722 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:49.276732 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/207b3254-36dd-450e-841a-bc771bf4ac3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8484f78-k2xzf\" (UID: \"207b3254-36dd-450e-841a-bc771bf4ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf" Oct 30 20:46:49.276932 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:49.276754 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/207b3254-36dd-450e-841a-bc771bf4ac3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8484f78-k2xzf\" (UID: \"207b3254-36dd-450e-841a-bc771bf4ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf" Oct 30 20:46:49.277350 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:49.277329 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/207b3254-36dd-450e-841a-bc771bf4ac3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8484f78-k2xzf\" (UID: \"207b3254-36dd-450e-841a-bc771bf4ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf" Oct 30 20:46:49.279256 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:49.279239 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/207b3254-36dd-450e-841a-bc771bf4ac3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8484f78-k2xzf\" (UID: \"207b3254-36dd-450e-841a-bc771bf4ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf" Oct 30 20:46:49.337096 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:49.337031 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7fz4\" (UniqueName: \"kubernetes.io/projected/207b3254-36dd-450e-841a-bc771bf4ac3a-kube-api-access-m7fz4\") pod \"maas-api-revoke-sub-a8484f78-k2xzf\" (UID: \"207b3254-36dd-450e-841a-bc771bf4ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf" Oct 30 20:46:49.404275 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:49.404247 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf" Oct 30 20:46:49.780362 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:49.780330 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf"] Oct 30 20:46:49.781295 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:46:49.781270 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207b3254_36dd_450e_841a_bc771bf4ac3a.slice/crio-e75eda0960d4342cad8aadc40de99d9d1b9312c885d9b2d07646fb82eec5f111 WatchSource:0}: Error finding container e75eda0960d4342cad8aadc40de99d9d1b9312c885d9b2d07646fb82eec5f111: Status 404 returned error can't find the container with id e75eda0960d4342cad8aadc40de99d9d1b9312c885d9b2d07646fb82eec5f111 Oct 30 20:46:50.661026 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:50.660995 5755 generic.go:356] "Generic (PLEG): container finished" podID="207b3254-36dd-450e-841a-bc771bf4ac3a" containerID="18627c0cdb1a71f7db6e385078123b709372171ee485256dc17d60952a683075" exitCode=0 Oct 30 20:46:50.661535 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:50.661050 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf" event={"ID":"207b3254-36dd-450e-841a-bc771bf4ac3a","Type":"ContainerDied","Data":"18627c0cdb1a71f7db6e385078123b709372171ee485256dc17d60952a683075"} Oct 30 20:46:50.661535 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:50.661093 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf" event={"ID":"207b3254-36dd-450e-841a-bc771bf4ac3a","Type":"ContainerStarted","Data":"e75eda0960d4342cad8aadc40de99d9d1b9312c885d9b2d07646fb82eec5f111"} Oct 30 20:46:51.815536 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:51.815509 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf" Oct 30 20:46:51.909851 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:51.909824 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/207b3254-36dd-450e-841a-bc771bf4ac3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/207b3254-36dd-450e-841a-bc771bf4ac3a-maas-api-xks-ca\") pod \"207b3254-36dd-450e-841a-bc771bf4ac3a\" (UID: \"207b3254-36dd-450e-841a-bc771bf4ac3a\") " Oct 30 20:46:51.910039 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:51.909883 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/207b3254-36dd-450e-841a-bc771bf4ac3a-kube-api-access-m7fz4\" (UniqueName: \"kubernetes.io/projected/207b3254-36dd-450e-841a-bc771bf4ac3a-kube-api-access-m7fz4\") pod \"207b3254-36dd-450e-841a-bc771bf4ac3a\" (UID: \"207b3254-36dd-450e-841a-bc771bf4ac3a\") " Oct 30 20:46:51.910086 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:51.910051 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/207b3254-36dd-450e-841a-bc771bf4ac3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/207b3254-36dd-450e-841a-bc771bf4ac3a-maas-api-openshift-ca\") pod \"207b3254-36dd-450e-841a-bc771bf4ac3a\" (UID: \"207b3254-36dd-450e-841a-bc771bf4ac3a\") " Oct 30 20:46:51.910455 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:51.910431 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207b3254-36dd-450e-841a-bc771bf4ac3a-maas-api-openshift-ca" pod "207b3254-36dd-450e-841a-bc771bf4ac3a" (UID: "207b3254-36dd-450e-841a-bc771bf4ac3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:46:51.912710 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:51.912671 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/207b3254-36dd-450e-841a-bc771bf4ac3a-maas-api-xks-ca" pod "207b3254-36dd-450e-841a-bc771bf4ac3a" (UID: "207b3254-36dd-450e-841a-bc771bf4ac3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:46:51.912710 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:51.912703 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207b3254-36dd-450e-841a-bc771bf4ac3a-kube-api-access-m7fz4" pod "207b3254-36dd-450e-841a-bc771bf4ac3a" (UID: "207b3254-36dd-450e-841a-bc771bf4ac3a"). InnerVolumeSpecName "kube-api-access-m7fz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:46:52.011833 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:52.011799 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/207b3254-36dd-450e-841a-bc771bf4ac3a-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:46:52.011833 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:52.011826 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/207b3254-36dd-450e-841a-bc771bf4ac3a-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:46:52.012037 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:52.011843 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7fz4\" (UniqueName: \"kubernetes.io/projected/207b3254-36dd-450e-841a-bc771bf4ac3a-kube-api-access-m7fz4\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:46:52.675303 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:52.675264 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf" event={"ID":"207b3254-36dd-450e-841a-bc771bf4ac3a","Type":"ContainerDied","Data":"e75eda0960d4342cad8aadc40de99d9d1b9312c885d9b2d07646fb82eec5f111"} Oct 30 20:46:52.675503 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:52.675314 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e75eda0960d4342cad8aadc40de99d9d1b9312c885d9b2d07646fb82eec5f111" Oct 30 20:46:52.675503 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:46:52.675277 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf" Oct 30 20:47:00.776542 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.776452 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56"] Oct 30 20:47:00.777024 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.777007 5755 cpu_manager.go:414] "containerMap: removing container" podUID="207b3254-36dd-450e-841a-bc771bf4ac3a" containerName="revoke-keys" Oct 30 20:47:00.777093 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.777025 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="207b3254-36dd-450e-841a-bc771bf4ac3a" containerName="revoke-keys" Oct 30 20:47:00.777141 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.777125 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="207b3254-36dd-450e-841a-bc771bf4ac3a" containerName="revoke-keys" Oct 30 20:47:00.780752 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.780732 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:00.826723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.826690 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4"] Oct 30 20:47:00.830516 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.830498 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:00.884258 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.884222 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56"] Oct 30 20:47:00.919097 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.919066 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:00.919097 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.919104 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:00.919363 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.919188 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:00.919363 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.919251 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmfrh\" (UniqueName: \"kubernetes.io/projected/cd8a8354-e543-4fe8-8385-de8b3ea8b417-kube-api-access-jmfrh\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:00.919363 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.919276 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:00.919363 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.919317 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:00.919363 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.919356 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:00.919634 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.919421 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:00.919634 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.919461 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:00.919706 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.919663 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:00.919761 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.919734 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:00.919899 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.919777 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:00.919899 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.919806 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:00.919899 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.919858 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd7gf\" (UniqueName: \"kubernetes.io/projected/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-kube-api-access-vd7gf\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:00.920080 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.919917 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:00.920080 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.919999 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:00.920080 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.920049 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:00.920237 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.920087 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:00.940944 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:00.940913 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4"] Oct 30 20:47:01.021304 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021264 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.021503 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021331 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.021503 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021370 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7gf\" (UniqueName: \"kubernetes.io/projected/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-kube-api-access-vd7gf\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.021503 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021452 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.021503 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021494 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.021739 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021533 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.021739 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021556 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.021739 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021618 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.021739 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021643 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.021739 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021704 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.021950 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021745 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmfrh\" (UniqueName: \"kubernetes.io/projected/cd8a8354-e543-4fe8-8385-de8b3ea8b417-kube-api-access-jmfrh\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.021950 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021772 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.021950 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021817 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.021950 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021856 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.021950 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021887 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.021950 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021926 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.022141 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021965 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.022141 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.021999 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.022492 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.022470 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.022755 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.022736 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.023059 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.023040 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.024324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.024047 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.024324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.024123 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.024324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.024238 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.024324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.024285 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.024640 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.024588 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.024739 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.024713 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.025022 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.024992 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.027320 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.027263 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.027654 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.027633 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.028707 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.028679 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.028900 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.028871 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.082271 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.082240 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7gf\" (UniqueName: \"kubernetes.io/projected/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-kube-api-access-vd7gf\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.082421 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.082253 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-qbz56\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.082421 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.082363 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.083007 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.082984 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmfrh\" (UniqueName: \"kubernetes.io/projected/cd8a8354-e543-4fe8-8385-de8b3ea8b417-kube-api-access-jmfrh\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-m22z4\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.093810 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.093784 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:01.144104 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.144063 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:01.528919 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.528889 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56"] Oct 30 20:47:01.530024 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:47:01.529998 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6b63f7_d73f_4b1a_8615_09d2f8ae2cc7.slice/crio-b1f5db68c6c3e47f49fffdf81c7e6735a026ab5c7de97bc3518cc32240a19a51 WatchSource:0}: Error finding container b1f5db68c6c3e47f49fffdf81c7e6735a026ab5c7de97bc3518cc32240a19a51: Status 404 returned error can't find the container with id b1f5db68c6c3e47f49fffdf81c7e6735a026ab5c7de97bc3518cc32240a19a51 Oct 30 20:47:01.532343 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.532306 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:01.532452 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.532404 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:01.532523 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.532452 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:01.584647 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.584622 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4"] Oct 30 20:47:01.587249 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:47:01.587225 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8a8354_e543_4fe8_8385_de8b3ea8b417.slice/crio-a73d3af427cf5fde4585046e987333b8e5cadaa109139d8bf6735b9ee71768e9 WatchSource:0}: Error finding container a73d3af427cf5fde4585046e987333b8e5cadaa109139d8bf6735b9ee71768e9: Status 404 returned error can't find the container with id a73d3af427cf5fde4585046e987333b8e5cadaa109139d8bf6735b9ee71768e9 Oct 30 20:47:01.590194 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.590162 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:01.590271 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.590224 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:01.590271 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.590252 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:01.731890 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.731847 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" event={"ID":"cd8a8354-e543-4fe8-8385-de8b3ea8b417","Type":"ContainerStarted","Data":"00772390dfe7987a298742b10a6f6bb0c36d13bc7946f895cab6a833d7d8a683"} Oct 30 20:47:01.732109 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.731918 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" event={"ID":"cd8a8354-e543-4fe8-8385-de8b3ea8b417","Type":"ContainerStarted","Data":"a73d3af427cf5fde4585046e987333b8e5cadaa109139d8bf6735b9ee71768e9"} Oct 30 20:47:01.733882 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.733857 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" event={"ID":"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7","Type":"ContainerStarted","Data":"ce4ff84590afad8ccf5ac2337fe4eaa809691d4f7b0a2855576b07b558a3ead6"} Oct 30 20:47:01.734026 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.733895 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" event={"ID":"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7","Type":"ContainerStarted","Data":"b1f5db68c6c3e47f49fffdf81c7e6735a026ab5c7de97bc3518cc32240a19a51"} Oct 30 20:47:01.856398 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.856288 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" podStartSLOduration=1.856273538 podStartE2EDuration="1.856273538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:01.853186093 +0000 UTC m=+1184.779743034" watchObservedRunningTime="2026-10-30 20:47:01.856273538 +0000 UTC m=+1184.782830452" Oct 30 20:47:01.977517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:01.977457 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" podStartSLOduration=1.9774435700000002 podStartE2EDuration="1.97744357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:01.973590942 +0000 UTC m=+1184.900147895" watchObservedRunningTime="2026-10-30 20:47:01.97744357 +0000 UTC m=+1184.904000482" Oct 30 20:47:02.094923 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:02.094890 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:02.096321 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:02.096297 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:47:02.096460 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:02.096342 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" podUID="ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:47:02.145134 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:02.145062 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:02.146485 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:02.146451 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:47:02.146620 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:02.146510 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" podUID="cd8a8354-e543-4fe8-8385-de8b3ea8b417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:47:03.096412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:03.095846 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:47:03.096412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:03.095913 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" podUID="ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:47:03.145067 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:03.145032 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:47:03.145243 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:03.145088 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" podUID="cd8a8354-e543-4fe8-8385-de8b3ea8b417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:47:04.114423 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:04.113538 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:47:04.114423 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:04.113612 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" podUID="ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:47:04.153875 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:04.153836 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:47:04.154161 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:04.154129 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" podUID="cd8a8354-e543-4fe8-8385-de8b3ea8b417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:47:05.094210 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:05.094161 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:47:05.094618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:05.094584 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" podUID="ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:47:05.146135 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:05.146078 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:47:05.146752 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:05.146696 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" podUID="cd8a8354-e543-4fe8-8385-de8b3ea8b417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:47:05.308070 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:05.308027 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:05.469483 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:05.453490 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:05.515412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:05.514501 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:05.589546 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:05.589496 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:05.750474 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:05.749512 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:05.827545 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:05.827494 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:05.907441 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:05.907368 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 20:47:05.977411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:05.966686 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:05.989558 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:05.989508 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:06.017616 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.017495 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:06.098787 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.098749 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:47:06.099067 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.099036 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" podUID="ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:47:06.114533 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.114483 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:06.147787 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.147752 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:47:06.148370 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.148342 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" podUID="cd8a8354-e543-4fe8-8385-de8b3ea8b417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:47:06.239528 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.239481 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:06.309797 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.309696 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:06.310361 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.310339 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:47:06.456987 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.456944 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:06.459173 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.458032 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:47:06.525412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.516487 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 20:47:06.525412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.516935 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:47:06.630596 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.630499 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:06.681415 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.631199 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:47:06.880750 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.880647 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:06.881330 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.881308 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:47:06.893056 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.893016 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 20:47:06.893610 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.893589 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:47:06.909653 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.909611 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:06.914121 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.914095 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:47:07.001410 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.996777 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:07.001410 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:06.997434 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:47:07.011649 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.011609 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 20:47:07.012277 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.012257 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:47:07.038840 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.038788 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:47:07.143464 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.082026 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:47:07.172443 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.168484 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:07.172443 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.169250 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:47:07.172443 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.169959 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:47:07.172443 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.170012 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" podUID="ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:47:07.172443 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.171234 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:47:07.172443 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.171273 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" podUID="cd8a8354-e543-4fe8-8385-de8b3ea8b417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:47:07.248440 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.248363 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:07.322613 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.322567 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:07.337462 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.337426 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:07.458022 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.457919 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:47:07.646285 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.646243 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 20:47:07.662499 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.662460 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:07.671498 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.670986 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:47:07.671498 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.671077 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:47:07.671498 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.671108 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:47:07.671498 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.671139 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:47:07.671498 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.671167 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:47:07.671498 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.671196 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:47:07.671498 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.671226 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:47:07.671498 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.671255 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:47:07.679252 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.677479 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:47:07.679252 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.677545 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:47:07.690826 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.690795 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:47:07.751777 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.750318 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:47:07.762700 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:07.762676 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:47:08.096925 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:08.096820 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:47:08.116411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:08.097159 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" podUID="ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:47:08.145122 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:08.145091 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:47:08.145330 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:08.145140 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" podUID="cd8a8354-e543-4fe8-8385-de8b3ea8b417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:47:09.094905 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:09.094874 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:47:09.095317 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:09.094939 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" podUID="ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:47:09.144702 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:09.144664 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:47:09.144875 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:09.144722 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" podUID="cd8a8354-e543-4fe8-8385-de8b3ea8b417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:47:10.308518 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:10.308478 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:10.309088 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:10.309064 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:10.309950 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:10.309928 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 20:47:10.522504 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:10.522476 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:10.522909 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:10.522882 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:10.523658 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:10.523637 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 20:47:11.255781 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:47:11.255748 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:47:15.243834 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:15.243799 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj"] Oct 30 20:47:15.247671 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:15.247652 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj" Oct 30 20:47:15.346857 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:15.346821 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd51c45-4e46-4841-acdb-073d7e3441fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3795a660-6dgjj\" (UID: \"dfd51c45-4e46-4841-acdb-073d7e3441fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj" Oct 30 20:47:15.347056 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:15.346939 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd51c45-4e46-4841-acdb-073d7e3441fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3795a660-6dgjj\" (UID: \"dfd51c45-4e46-4841-acdb-073d7e3441fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj" Oct 30 20:47:15.347115 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:15.347087 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rj7r\" (UniqueName: \"kubernetes.io/projected/dfd51c45-4e46-4841-acdb-073d7e3441fc-kube-api-access-6rj7r\") pod \"maas-api-revoke-sub-3795a660-6dgjj\" (UID: \"dfd51c45-4e46-4841-acdb-073d7e3441fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj" Oct 30 20:47:15.351731 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:15.351707 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:47:15.351870 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:15.351797 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:47:15.367202 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:15.367166 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj"] Oct 30 20:47:15.448688 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:15.448652 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd51c45-4e46-4841-acdb-073d7e3441fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3795a660-6dgjj\" (UID: \"dfd51c45-4e46-4841-acdb-073d7e3441fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj" Oct 30 20:47:15.448844 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:15.448707 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd51c45-4e46-4841-acdb-073d7e3441fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3795a660-6dgjj\" (UID: \"dfd51c45-4e46-4841-acdb-073d7e3441fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj" Oct 30 20:47:15.448844 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:15.448793 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rj7r\" (UniqueName: \"kubernetes.io/projected/dfd51c45-4e46-4841-acdb-073d7e3441fc-kube-api-access-6rj7r\") pod \"maas-api-revoke-sub-3795a660-6dgjj\" (UID: \"dfd51c45-4e46-4841-acdb-073d7e3441fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj" Oct 30 20:47:15.449356 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:15.449330 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd51c45-4e46-4841-acdb-073d7e3441fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3795a660-6dgjj\" (UID: \"dfd51c45-4e46-4841-acdb-073d7e3441fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj" Oct 30 20:47:15.451248 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:15.451225 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd51c45-4e46-4841-acdb-073d7e3441fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3795a660-6dgjj\" (UID: \"dfd51c45-4e46-4841-acdb-073d7e3441fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj" Oct 30 20:47:15.507033 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:15.506964 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rj7r\" (UniqueName: \"kubernetes.io/projected/dfd51c45-4e46-4841-acdb-073d7e3441fc-kube-api-access-6rj7r\") pod \"maas-api-revoke-sub-3795a660-6dgjj\" (UID: \"dfd51c45-4e46-4841-acdb-073d7e3441fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj" Oct 30 20:47:15.562231 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:15.562198 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj" Oct 30 20:47:15.945846 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:15.945815 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj"] Oct 30 20:47:15.947509 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:47:15.947472 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd51c45_4e46_4841_acdb_073d7e3441fc.slice/crio-76fcf5f9805dbf1525562896b8da0beb369b03228904edd0d2cd409af6a46658 WatchSource:0}: Error finding container 76fcf5f9805dbf1525562896b8da0beb369b03228904edd0d2cd409af6a46658: Status 404 returned error can't find the container with id 76fcf5f9805dbf1525562896b8da0beb369b03228904edd0d2cd409af6a46658 Oct 30 20:47:16.271861 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:16.271828 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj" event={"ID":"dfd51c45-4e46-4841-acdb-073d7e3441fc","Type":"ContainerStarted","Data":"76fcf5f9805dbf1525562896b8da0beb369b03228904edd0d2cd409af6a46658"} Oct 30 20:47:17.279355 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:17.279312 5755 generic.go:356] "Generic (PLEG): container finished" podID="dfd51c45-4e46-4841-acdb-073d7e3441fc" containerID="1efbfab64a5b6e351e54ed0794343942ae5b70255f30cd3efbc6c3fa08c9582d" exitCode=0 Oct 30 20:47:17.279815 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:17.279421 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj" event={"ID":"dfd51c45-4e46-4841-acdb-073d7e3441fc","Type":"ContainerDied","Data":"1efbfab64a5b6e351e54ed0794343942ae5b70255f30cd3efbc6c3fa08c9582d"} Oct 30 20:47:18.051865 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:18.051829 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj"] Oct 30 20:47:18.056446 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:18.056421 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a5961ce-pjvfj"] Oct 30 20:47:18.435704 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:18.435682 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj" Oct 30 20:47:18.487635 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:18.487596 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd51c45-4e46-4841-acdb-073d7e3441fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd51c45-4e46-4841-acdb-073d7e3441fc-maas-api-xks-ca\") pod \"dfd51c45-4e46-4841-acdb-073d7e3441fc\" (UID: \"dfd51c45-4e46-4841-acdb-073d7e3441fc\") " Oct 30 20:47:18.487785 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:18.487683 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd51c45-4e46-4841-acdb-073d7e3441fc-kube-api-access-6rj7r\" (UniqueName: \"kubernetes.io/projected/dfd51c45-4e46-4841-acdb-073d7e3441fc-kube-api-access-6rj7r\") pod \"dfd51c45-4e46-4841-acdb-073d7e3441fc\" (UID: \"dfd51c45-4e46-4841-acdb-073d7e3441fc\") " Oct 30 20:47:18.487785 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:18.487781 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd51c45-4e46-4841-acdb-073d7e3441fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd51c45-4e46-4841-acdb-073d7e3441fc-maas-api-openshift-ca\") pod \"dfd51c45-4e46-4841-acdb-073d7e3441fc\" (UID: \"dfd51c45-4e46-4841-acdb-073d7e3441fc\") " Oct 30 20:47:18.488129 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:18.488104 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd51c45-4e46-4841-acdb-073d7e3441fc-maas-api-openshift-ca" pod "dfd51c45-4e46-4841-acdb-073d7e3441fc" (UID: "dfd51c45-4e46-4841-acdb-073d7e3441fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:18.489880 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:18.489856 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd51c45-4e46-4841-acdb-073d7e3441fc-maas-api-xks-ca" pod "dfd51c45-4e46-4841-acdb-073d7e3441fc" (UID: "dfd51c45-4e46-4841-acdb-073d7e3441fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:18.489960 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:18.489910 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd51c45-4e46-4841-acdb-073d7e3441fc-kube-api-access-6rj7r" pod "dfd51c45-4e46-4841-acdb-073d7e3441fc" (UID: "dfd51c45-4e46-4841-acdb-073d7e3441fc"). InnerVolumeSpecName "kube-api-access-6rj7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:18.589439 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:18.589349 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd51c45-4e46-4841-acdb-073d7e3441fc-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:18.589439 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:18.589374 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rj7r\" (UniqueName: \"kubernetes.io/projected/dfd51c45-4e46-4841-acdb-073d7e3441fc-kube-api-access-6rj7r\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:18.589439 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:18.589417 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd51c45-4e46-4841-acdb-073d7e3441fc-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:19.212660 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:19.212628 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5396a0db-8f86-4fee-b99d-f629526be15d" path="/var/lib/kubelet/pods/5396a0db-8f86-4fee-b99d-f629526be15d/volumes" Oct 30 20:47:19.293063 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:19.293039 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj" Oct 30 20:47:19.293188 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:19.293087 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj" event={"ID":"dfd51c45-4e46-4841-acdb-073d7e3441fc","Type":"ContainerDied","Data":"76fcf5f9805dbf1525562896b8da0beb369b03228904edd0d2cd409af6a46658"} Oct 30 20:47:19.293188 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:19.293133 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76fcf5f9805dbf1525562896b8da0beb369b03228904edd0d2cd409af6a46658" Oct 30 20:47:30.651829 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.651783 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk"] Oct 30 20:47:30.652559 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.652543 5755 cpu_manager.go:414] "containerMap: removing container" podUID="dfd51c45-4e46-4841-acdb-073d7e3441fc" containerName="revoke-keys" Oct 30 20:47:30.652627 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.652562 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd51c45-4e46-4841-acdb-073d7e3441fc" containerName="revoke-keys" Oct 30 20:47:30.652714 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.652690 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd51c45-4e46-4841-acdb-073d7e3441fc" containerName="revoke-keys" Oct 30 20:47:30.656434 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.656410 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.762956 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.762927 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f6ebf5-openshift-default-dockercfg-hsckc\"" Oct 30 20:47:30.777797 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.767106 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk"] Oct 30 20:47:30.827764 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.827734 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.827918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.827781 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aa5e94-167d-44d7-b423-54914f1b2182-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.827918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.827811 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7jk2\" (UniqueName: \"kubernetes.io/projected/91aa5e94-167d-44d7-b423-54914f1b2182-kube-api-access-h7jk2\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.827918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.827899 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.828021 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.827952 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.828021 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.827997 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aa5e94-167d-44d7-b423-54914f1b2182-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.828099 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.828064 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91aa5e94-167d-44d7-b423-54914f1b2182-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.828133 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.828118 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.828166 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.828140 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.929504 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.929411 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aa5e94-167d-44d7-b423-54914f1b2182-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.929504 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.929476 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7jk2\" (UniqueName: \"kubernetes.io/projected/91aa5e94-167d-44d7-b423-54914f1b2182-kube-api-access-h7jk2\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.929722 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.929515 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.929722 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.929671 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.929828 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.929771 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aa5e94-167d-44d7-b423-54914f1b2182-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.929886 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.929846 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91aa5e94-167d-44d7-b423-54914f1b2182-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.929944 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.929924 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.929997 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.929936 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.929997 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.929984 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.930109 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.930079 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.930255 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.930225 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aa5e94-167d-44d7-b423-54914f1b2182-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.930372 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.930255 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.930478 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.930456 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.930573 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.930503 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.932426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.932379 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.932526 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.932489 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aa5e94-167d-44d7-b423-54914f1b2182-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.988334 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.988306 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91aa5e94-167d-44d7-b423-54914f1b2182-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:30.988487 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:30.988421 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7jk2\" (UniqueName: \"kubernetes.io/projected/91aa5e94-167d-44d7-b423-54914f1b2182-kube-api-access-h7jk2\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:31.058219 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.058189 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h"] Oct 30 20:47:31.060832 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.060810 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a169dad7-pkz5h"] Oct 30 20:47:31.212529 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.212462 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de445e8-8a06-4000-bfb9-be20213a0b12" path="/var/lib/kubelet/pods/3de445e8-8a06-4000-bfb9-be20213a0b12/volumes" Oct 30 20:47:31.270907 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.270886 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:31.348734 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.348059 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf"] Oct 30 20:47:31.354142 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.354112 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.409829 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.409799 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf"] Oct 30 20:47:31.537064 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.537034 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-podinfo\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.537230 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.537091 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-envoy\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.537230 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.537181 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4rjf\" (UniqueName: \"kubernetes.io/projected/81547e38-6ed0-4a54-bdec-8972a18f2f1a-kube-api-access-h4rjf\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.537355 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.537251 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-workload-certs\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.537432 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.537353 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-workload-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.537514 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.537458 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-data\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.537514 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.537499 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-token\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.537600 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.537575 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-credential-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.537637 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.537608 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istiod-ca-cert\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.638744 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.638719 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-podinfo\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.638897 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.638773 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-envoy\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.638897 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.638801 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4rjf\" (UniqueName: \"kubernetes.io/projected/81547e38-6ed0-4a54-bdec-8972a18f2f1a-kube-api-access-h4rjf\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.638897 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.638851 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-workload-certs\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.638897 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.638881 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-workload-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.639083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.638914 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-data\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.639083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.638942 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-token\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.639083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.638966 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-credential-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.639083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.638992 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istiod-ca-cert\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.639347 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.639318 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-workload-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.639489 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.639355 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-workload-certs\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.639701 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.639679 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-credential-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.639930 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.639905 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-data\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.639994 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.639956 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istiod-ca-cert\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.641347 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.641322 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-envoy\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.641451 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.641437 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-podinfo\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.700797 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.700779 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4rjf\" (UniqueName: \"kubernetes.io/projected/81547e38-6ed0-4a54-bdec-8972a18f2f1a-kube-api-access-h4rjf\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.702631 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.702612 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-token\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:31.711147 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.711128 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk"] Oct 30 20:47:31.713997 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:47:31.713973 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91aa5e94_167d_44d7_b423_54914f1b2182.slice/crio-0756d61c4bd1857ae514412628a2cddccf628cc7b4a260f96a39999fcb14a87d WatchSource:0}: Error finding container 0756d61c4bd1857ae514412628a2cddccf628cc7b4a260f96a39999fcb14a87d: Status 404 returned error can't find the container with id 0756d61c4bd1857ae514412628a2cddccf628cc7b4a260f96a39999fcb14a87d Oct 30 20:47:31.716367 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.716333 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:31.716458 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.716422 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:31.716458 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.716452 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:31.767695 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.767670 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j"] Oct 30 20:47:31.771546 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.771527 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:31.837986 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.837911 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt"] Oct 30 20:47:31.842152 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.842127 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:31.848427 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.846880 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j"] Oct 30 20:47:31.902807 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.902782 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt"] Oct 30 20:47:31.949030 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949000 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx25m\" (UniqueName: \"kubernetes.io/projected/1ddd9841-3766-4d6a-984b-1410f58eba31-kube-api-access-zx25m\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:31.949181 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949043 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:31.949181 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949164 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:31.949307 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949189 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:31.949307 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949218 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:31.949307 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949239 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:31.949503 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949375 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:31.949503 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949446 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:31.949667 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949503 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:31.949667 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949536 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56r5p\" (UniqueName: \"kubernetes.io/projected/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-kube-api-access-56r5p\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:31.949667 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949579 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:31.949823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949666 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ddd9841-3766-4d6a-984b-1410f58eba31-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:31.949823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949719 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:31.949823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949778 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:31.949823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949804 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:31.950006 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949909 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:31.950006 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.949941 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:31.950081 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.950036 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:31.970555 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:31.970509 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:32.051587 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.051557 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.051819 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.051612 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.051819 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.051660 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.051819 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.051688 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56r5p\" (UniqueName: \"kubernetes.io/projected/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-kube-api-access-56r5p\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.051819 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.051740 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.051819 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.051788 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ddd9841-3766-4d6a-984b-1410f58eba31-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.052874 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.051908 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.052874 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.051968 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.052874 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.051990 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.052874 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.052089 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.052874 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.052140 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.052874 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.052193 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.052874 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.052221 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx25m\" (UniqueName: \"kubernetes.io/projected/1ddd9841-3766-4d6a-984b-1410f58eba31-kube-api-access-zx25m\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.052874 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.052249 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.052874 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.052331 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.052874 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.052354 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.057223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.052961 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.057223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.053004 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.057223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.053696 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.057223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.053829 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.057223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.053998 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.057223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.054409 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.057223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.054659 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.057223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.054671 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.057223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.055138 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.057223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.055438 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.057223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.055558 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.057223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.056182 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ddd9841-3766-4d6a-984b-1410f58eba31-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.058939 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.058891 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.060326 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.060280 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.060817 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.060791 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.061208 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.061187 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.114769 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.114688 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56r5p\" (UniqueName: \"kubernetes.io/projected/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-kube-api-access-56r5p\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.115063 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.115037 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.115201 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.115184 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx25m\" (UniqueName: \"kubernetes.io/projected/1ddd9841-3766-4d6a-984b-1410f58eba31-kube-api-access-zx25m\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.115781 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.115762 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.194347 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.194317 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:32.363191 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.363163 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf"] Oct 30 20:47:32.365506 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:47:32.365445 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81547e38_6ed0_4a54_bdec_8972a18f2f1a.slice/crio-a4ebbac5a7d67ac63c18a6a686d259e29204f3a0f1d9fe802d1f575645d8f67e WatchSource:0}: Error finding container a4ebbac5a7d67ac63c18a6a686d259e29204f3a0f1d9fe802d1f575645d8f67e: Status 404 returned error can't find the container with id a4ebbac5a7d67ac63c18a6a686d259e29204f3a0f1d9fe802d1f575645d8f67e Oct 30 20:47:32.371977 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.371935 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:32.372080 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.372015 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:32.372080 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.372059 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:32.377120 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.377095 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" event={"ID":"81547e38-6ed0-4a54-bdec-8972a18f2f1a","Type":"ContainerStarted","Data":"a4ebbac5a7d67ac63c18a6a686d259e29204f3a0f1d9fe802d1f575645d8f67e"} Oct 30 20:47:32.378662 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.378639 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" event={"ID":"91aa5e94-167d-44d7-b423-54914f1b2182","Type":"ContainerStarted","Data":"87a13f6a2204e3b825c87c85dbfa04b25b704823ce3d6f8d2e7e377b7e04e049"} Oct 30 20:47:32.378763 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.378680 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" event={"ID":"91aa5e94-167d-44d7-b423-54914f1b2182","Type":"ContainerStarted","Data":"0756d61c4bd1857ae514412628a2cddccf628cc7b4a260f96a39999fcb14a87d"} Oct 30 20:47:32.386950 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.386932 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:32.502133 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.502083 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" podStartSLOduration=2.502068709 podStartE2EDuration="2.502068709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:32.497244147 +0000 UTC m=+1215.423801090" watchObservedRunningTime="2026-10-30 20:47:32.502068709 +0000 UTC m=+1215.428625623" Oct 30 20:47:32.589520 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.589480 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:32.589729 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.589709 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:32.589871 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.589842 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:32.590848 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.590826 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt"] Oct 30 20:47:32.779886 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.779853 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j"] Oct 30 20:47:32.781219 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:47:32.781192 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ddd9841_3766_4d6a_984b_1410f58eba31.slice/crio-d905b990f0a8d76bdd1f411c7c50066d2ead4557760c7a9f3960cd46602956ea WatchSource:0}: Error finding container d905b990f0a8d76bdd1f411c7c50066d2ead4557760c7a9f3960cd46602956ea: Status 404 returned error can't find the container with id d905b990f0a8d76bdd1f411c7c50066d2ead4557760c7a9f3960cd46602956ea Oct 30 20:47:32.784098 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.784054 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:32.784210 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.784144 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:32.784210 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:32.784186 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:33.273294 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:33.273267 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:33.280417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:33.273632 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:47:33.280708 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:33.280669 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" podUID="91aa5e94-167d-44d7-b423-54914f1b2182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 20:47:33.436446 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:33.436405 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" event={"ID":"81547e38-6ed0-4a54-bdec-8972a18f2f1a","Type":"ContainerStarted","Data":"d0d218d7870b7fa938e4a1cb95568c4452b400f577f8e53ce85b3268543a7c09"} Oct 30 20:47:33.440610 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:33.440575 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" event={"ID":"b830f9c4-8e19-4ee1-9bd3-e41b667a9122","Type":"ContainerStarted","Data":"60136f3eb20c6b89dc276d41b6a4e1310612431c8428e20d7fb0767373993c0f"} Oct 30 20:47:33.440773 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:33.440630 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" event={"ID":"b830f9c4-8e19-4ee1-9bd3-e41b667a9122","Type":"ContainerStarted","Data":"9b49caf69e00b5462fde5fad6a8c0260e12e6a418b750d3debb14d5a4e0011ce"} Oct 30 20:47:33.445839 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:33.445810 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" event={"ID":"1ddd9841-3766-4d6a-984b-1410f58eba31","Type":"ContainerStarted","Data":"b7cb72ef22895d1aed2ed05360b16e18d66f8256b4dd51cd16e45cf76b338b1a"} Oct 30 20:47:33.446048 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:33.446031 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" event={"ID":"1ddd9841-3766-4d6a-984b-1410f58eba31","Type":"ContainerStarted","Data":"d905b990f0a8d76bdd1f411c7c50066d2ead4557760c7a9f3960cd46602956ea"} Oct 30 20:47:33.719507 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:33.718802 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" podStartSLOduration=2.718782191 podStartE2EDuration="2.718782191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:33.718036259 +0000 UTC m=+1216.644593199" watchObservedRunningTime="2026-10-30 20:47:33.718782191 +0000 UTC m=+1216.645339108" Oct 30 20:47:33.736426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:33.724263 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" podStartSLOduration=2.724230837 podStartE2EDuration="2.724230837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:33.597550023 +0000 UTC m=+1216.524106956" watchObservedRunningTime="2026-10-30 20:47:33.724230837 +0000 UTC m=+1216.650787753" Oct 30 20:47:33.996468 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:33.993684 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:33.996468 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:33.993832 5755 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:47:33.996468 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:33.993889 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" podUID="81547e38-6ed0-4a54-bdec-8972a18f2f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:47:34.053543 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.053478 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" podStartSLOduration=3.053450435 podStartE2EDuration="3.053450435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:33.859792188 +0000 UTC m=+1216.786349124" watchObservedRunningTime="2026-10-30 20:47:34.053450435 +0000 UTC m=+1216.980007353" Oct 30 20:47:34.109412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.088452 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7"] Oct 30 20:47:34.146568 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.146531 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9483bf5-fgmx7"] Oct 30 20:47:34.213413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.202796 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:34.213413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.206505 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:47:34.213413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.206561 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" podUID="b830f9c4-8e19-4ee1-9bd3-e41b667a9122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 20:47:34.284468 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.277894 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:47:34.284468 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.277982 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" podUID="91aa5e94-167d-44d7-b423-54914f1b2182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 20:47:34.305415 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.295613 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4"] Oct 30 20:47:34.318229 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.318199 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" Oct 30 20:47:34.391991 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.388723 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:34.391991 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.389012 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:47:34.391991 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.390051 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:47:34.391991 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.391873 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:47:34.391991 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.391916 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" podUID="1ddd9841-3766-4d6a-984b-1410f58eba31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:47:34.450415 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.435688 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15171cf4-7349-4232-a057-2d28352faae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37f82c1a-nx8x4\" (UID: \"15171cf4-7349-4232-a057-2d28352faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" Oct 30 20:47:34.646472 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.645728 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15171cf4-7349-4232-a057-2d28352faae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37f82c1a-nx8x4\" (UID: \"15171cf4-7349-4232-a057-2d28352faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" Oct 30 20:47:34.646472 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.645859 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssdff\" (UniqueName: \"kubernetes.io/projected/15171cf4-7349-4232-a057-2d28352faae1-kube-api-access-ssdff\") pod \"maas-api-revoke-sub-37f82c1a-nx8x4\" (UID: \"15171cf4-7349-4232-a057-2d28352faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" Oct 30 20:47:34.687413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.676289 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4"] Oct 30 20:47:34.751413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.750448 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15171cf4-7349-4232-a057-2d28352faae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37f82c1a-nx8x4\" (UID: \"15171cf4-7349-4232-a057-2d28352faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" Oct 30 20:47:34.751413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.750643 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssdff\" (UniqueName: \"kubernetes.io/projected/15171cf4-7349-4232-a057-2d28352faae1-kube-api-access-ssdff\") pod \"maas-api-revoke-sub-37f82c1a-nx8x4\" (UID: \"15171cf4-7349-4232-a057-2d28352faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" Oct 30 20:47:34.751413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.750747 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15171cf4-7349-4232-a057-2d28352faae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37f82c1a-nx8x4\" (UID: \"15171cf4-7349-4232-a057-2d28352faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" Oct 30 20:47:34.784413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.770924 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15171cf4-7349-4232-a057-2d28352faae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37f82c1a-nx8x4\" (UID: \"15171cf4-7349-4232-a057-2d28352faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" Oct 30 20:47:34.806886 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.806852 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15171cf4-7349-4232-a057-2d28352faae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37f82c1a-nx8x4\" (UID: \"15171cf4-7349-4232-a057-2d28352faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" Oct 30 20:47:34.870412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.868698 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssdff\" (UniqueName: \"kubernetes.io/projected/15171cf4-7349-4232-a057-2d28352faae1-kube-api-access-ssdff\") pod \"maas-api-revoke-sub-37f82c1a-nx8x4\" (UID: \"15171cf4-7349-4232-a057-2d28352faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" Oct 30 20:47:34.941462 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:34.940619 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" Oct 30 20:47:35.067716 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.067669 5755 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:47:35.068698 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.068645 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" podUID="81547e38-6ed0-4a54-bdec-8972a18f2f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:47:35.198913 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.198819 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:47:35.199228 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.199159 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" podUID="b830f9c4-8e19-4ee1-9bd3-e41b667a9122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 20:47:35.268374 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.263308 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf3a0d3-0027-4206-8b22-2f91f716329c" path="/var/lib/kubelet/pods/8cf3a0d3-0027-4206-8b22-2f91f716329c/volumes" Oct 30 20:47:35.278445 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.276781 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:47:35.278445 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.276837 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" podUID="91aa5e94-167d-44d7-b423-54914f1b2182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 20:47:35.367136 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.361503 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:35.368496 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.368453 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:35.389935 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.389809 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:47:35.389935 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.389865 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" podUID="1ddd9841-3766-4d6a-984b-1410f58eba31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:47:35.449347 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.448860 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:35.673413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.661239 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:35.673413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.661496 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 20:47:35.673413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.661776 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 20:47:35.715466 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.713630 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:47:35.715466 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.714895 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:35.810422 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.717028 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 20:47:35.870730 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:35.815495 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:36.022578 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.016756 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4"] Oct 30 20:47:36.084380 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.070327 5755 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:47:36.084380 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.070418 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" podUID="81547e38-6ed0-4a54-bdec-8972a18f2f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:47:36.336924 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.311941 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:47:36.336924 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.312011 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" podUID="b830f9c4-8e19-4ee1-9bd3-e41b667a9122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 20:47:36.336924 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.312115 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:47:36.336924 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.312139 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" podUID="91aa5e94-167d-44d7-b423-54914f1b2182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 20:47:36.424564 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.424510 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:36.454463 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.454413 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:47:36.660464 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.456407 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:47:36.739524 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.739474 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" podUID="1ddd9841-3766-4d6a-984b-1410f58eba31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:47:36.739802 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.726540 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:47:36.748483 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.726496 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:47:36.875411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.791450 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:36.875411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.792451 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:47:36.875411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.845693 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:36.875411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.846441 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:47:36.934588 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.890414 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:47:36.934588 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.890868 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 20:47:36.934588 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.891241 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:47:36.934588 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.891328 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:47:36.934588 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.932019 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 20:47:36.934588 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.932356 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 20:47:36.942259 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.942234 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:47:36.961435 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:36.961411 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:47:37.068423 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.067035 5755 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:47:37.068618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.068528 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" podUID="81547e38-6ed0-4a54-bdec-8972a18f2f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:47:37.334414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.333027 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:47:37.334414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.333083 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" podUID="b830f9c4-8e19-4ee1-9bd3-e41b667a9122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 20:47:37.334414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.333164 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:47:37.334414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.333185 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" podUID="91aa5e94-167d-44d7-b423-54914f1b2182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 20:47:37.334414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.333231 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 20:47:37.358582 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.341872 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:37.430533 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.430478 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:37.554561 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.554520 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:47:37.607747 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.568855 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:37.607994 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.607959 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" podUID="1ddd9841-3766-4d6a-984b-1410f58eba31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:47:37.713412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.712938 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" event={"ID":"15171cf4-7349-4232-a057-2d28352faae1","Type":"ContainerStarted","Data":"a5d4cb3cbfffbcb29b2e302d2aa76240bda9704a49bf68db30fbe9402e03e4fe"} Oct 30 20:47:37.713412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.713040 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:47:37.728750 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.728707 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:37.748412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.740892 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:37.749006 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.748979 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:47:37.795509 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.795465 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 20:47:37.796822 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.796796 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:47:37.887464 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.883596 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:37.887464 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.883910 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 20:47:37.930744 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.916004 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 20:47:37.930744 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.918164 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:47:37.937714 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.935136 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:37.968606 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.961210 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:37.982991 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:37.982953 5755 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:47:38.000625 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.000589 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" podUID="81547e38-6ed0-4a54-bdec-8972a18f2f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:47:38.233413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.226229 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:47:38.233413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.227044 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:47:38.233413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.227086 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" podUID="b830f9c4-8e19-4ee1-9bd3-e41b667a9122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 20:47:38.323450 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.284838 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:47:38.323450 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.284895 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" podUID="91aa5e94-167d-44d7-b423-54914f1b2182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 20:47:38.363411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.341670 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:47:38.363411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.342051 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 20:47:38.363411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.342219 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:47:38.363411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.343084 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:47:38.443459 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.442862 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:47:38.443459 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.442922 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" podUID="1ddd9841-3766-4d6a-984b-1410f58eba31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:47:38.455505 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.451469 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:38.569420 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.565572 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:38.608517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.608470 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:38.615418 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.609091 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:47:38.759584 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.759537 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:38.797074 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.797030 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 20:47:38.797819 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.760490 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:47:38.798602 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.798584 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:47:38.902413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.882791 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" event={"ID":"15171cf4-7349-4232-a057-2d28352faae1","Type":"ContainerStarted","Data":"589bec91b575f833c2fb11638354dbc85682aeccf0fa76047dba9f043a90c218"} Oct 30 20:47:38.999412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.922957 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 20:47:38.999412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.982211 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:38.999412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.976530 5755 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:47:38.999412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:38.982473 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" podUID="81547e38-6ed0-4a54-bdec-8972a18f2f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:47:39.206788 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.206752 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:47:39.269018 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.207013 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" podUID="b830f9c4-8e19-4ee1-9bd3-e41b667a9122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 20:47:39.370518 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.283976 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:47:39.370518 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.284033 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" podUID="91aa5e94-167d-44d7-b423-54914f1b2182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 20:47:39.432414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.402787 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:47:39.432414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.402854 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:47:39.432414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.406150 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:47:39.432414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.406197 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" podUID="1ddd9841-3766-4d6a-984b-1410f58eba31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:47:39.542461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.457419 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:39.602684 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.602634 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:39.615424 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.603279 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:47:39.662061 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.661980 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:47:39.849679 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.849625 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:39.850360 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.850325 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 20:47:39.851111 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.851088 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:47:39.868599 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.858202 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:47:39.949549 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.949471 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:47:39.961644 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.961615 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:47:39.990461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.982481 5755 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:47:39.990461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:39.982551 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" podUID="81547e38-6ed0-4a54-bdec-8972a18f2f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:47:40.017039 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:40.016998 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:47:40.017365 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:40.017344 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:47:40.018739 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:40.018719 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:47:40.021034 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:40.021008 5755 generic.go:356] "Generic (PLEG): container finished" podID="15171cf4-7349-4232-a057-2d28352faae1" containerID="589bec91b575f833c2fb11638354dbc85682aeccf0fa76047dba9f043a90c218" exitCode=0 Oct 30 20:47:40.021987 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:40.021963 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" event={"ID":"15171cf4-7349-4232-a057-2d28352faae1","Type":"ContainerDied","Data":"589bec91b575f833c2fb11638354dbc85682aeccf0fa76047dba9f043a90c218"} Oct 30 20:47:40.196797 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:40.196745 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:47:40.197057 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:40.197029 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" podUID="b830f9c4-8e19-4ee1-9bd3-e41b667a9122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 20:47:40.271938 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:40.271887 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:47:40.272219 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:40.272188 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" podUID="91aa5e94-167d-44d7-b423-54914f1b2182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 20:47:40.391864 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:40.388809 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:47:40.391864 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:40.388874 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" podUID="1ddd9841-3766-4d6a-984b-1410f58eba31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:47:40.795116 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:40.794266 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" podStartSLOduration=6.794245468 podStartE2EDuration="6.794245468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:40.647848237 +0000 UTC m=+1223.574405154" watchObservedRunningTime="2026-10-30 20:47:40.794245468 +0000 UTC m=+1223.720802385" Oct 30 20:47:41.113593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.113554 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f"] Oct 30 20:47:41.132850 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.132818 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f" Oct 30 20:47:41.161624 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.161594 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f"] Oct 30 20:47:41.177979 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.177937 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2f2d\" (UniqueName: \"kubernetes.io/projected/25fc72ef-aea6-44a7-9077-40ecd86c99a2-kube-api-access-q2f2d\") pod \"maas-api-revoke-sub-8eee1174-2wk8f\" (UID: \"25fc72ef-aea6-44a7-9077-40ecd86c99a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f" Oct 30 20:47:41.178308 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.178285 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25fc72ef-aea6-44a7-9077-40ecd86c99a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eee1174-2wk8f\" (UID: \"25fc72ef-aea6-44a7-9077-40ecd86c99a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f" Oct 30 20:47:41.178602 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.178570 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25fc72ef-aea6-44a7-9077-40ecd86c99a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eee1174-2wk8f\" (UID: \"25fc72ef-aea6-44a7-9077-40ecd86c99a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f" Oct 30 20:47:41.314926 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.283167 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25fc72ef-aea6-44a7-9077-40ecd86c99a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eee1174-2wk8f\" (UID: \"25fc72ef-aea6-44a7-9077-40ecd86c99a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f" Oct 30 20:47:41.314926 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.283321 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2f2d\" (UniqueName: \"kubernetes.io/projected/25fc72ef-aea6-44a7-9077-40ecd86c99a2-kube-api-access-q2f2d\") pod \"maas-api-revoke-sub-8eee1174-2wk8f\" (UID: \"25fc72ef-aea6-44a7-9077-40ecd86c99a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f" Oct 30 20:47:41.314926 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.283473 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25fc72ef-aea6-44a7-9077-40ecd86c99a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eee1174-2wk8f\" (UID: \"25fc72ef-aea6-44a7-9077-40ecd86c99a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f" Oct 30 20:47:41.314926 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.284426 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25fc72ef-aea6-44a7-9077-40ecd86c99a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eee1174-2wk8f\" (UID: \"25fc72ef-aea6-44a7-9077-40ecd86c99a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f" Oct 30 20:47:41.314926 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.300716 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:41.314926 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.301848 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:41.322878 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.320792 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:47:41.322878 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:47:41.320992 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:47:41.348158 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.348117 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2f2d\" (UniqueName: \"kubernetes.io/projected/25fc72ef-aea6-44a7-9077-40ecd86c99a2-kube-api-access-q2f2d\") pod \"maas-api-revoke-sub-8eee1174-2wk8f\" (UID: \"25fc72ef-aea6-44a7-9077-40ecd86c99a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f" Oct 30 20:47:41.362413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.356826 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25fc72ef-aea6-44a7-9077-40ecd86c99a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eee1174-2wk8f\" (UID: \"25fc72ef-aea6-44a7-9077-40ecd86c99a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f" Oct 30 20:47:41.410721 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.410693 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:47:41.411378 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.411340 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" podUID="1ddd9841-3766-4d6a-984b-1410f58eba31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:47:41.462941 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.462915 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" Oct 30 20:47:41.471257 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.471226 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f" Oct 30 20:47:41.538535 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.537478 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15171cf4-7349-4232-a057-2d28352faae1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15171cf4-7349-4232-a057-2d28352faae1-maas-api-openshift-ca\") pod \"15171cf4-7349-4232-a057-2d28352faae1\" (UID: \"15171cf4-7349-4232-a057-2d28352faae1\") " Oct 30 20:47:41.538535 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.537607 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15171cf4-7349-4232-a057-2d28352faae1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15171cf4-7349-4232-a057-2d28352faae1-maas-api-xks-ca\") pod \"15171cf4-7349-4232-a057-2d28352faae1\" (UID: \"15171cf4-7349-4232-a057-2d28352faae1\") " Oct 30 20:47:41.538535 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.537709 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15171cf4-7349-4232-a057-2d28352faae1-kube-api-access-ssdff\" (UniqueName: \"kubernetes.io/projected/15171cf4-7349-4232-a057-2d28352faae1-kube-api-access-ssdff\") pod \"15171cf4-7349-4232-a057-2d28352faae1\" (UID: \"15171cf4-7349-4232-a057-2d28352faae1\") " Oct 30 20:47:41.541454 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.538892 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15171cf4-7349-4232-a057-2d28352faae1-maas-api-openshift-ca" pod "15171cf4-7349-4232-a057-2d28352faae1" (UID: "15171cf4-7349-4232-a057-2d28352faae1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:41.545424 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.544166 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15171cf4-7349-4232-a057-2d28352faae1-maas-api-xks-ca" pod "15171cf4-7349-4232-a057-2d28352faae1" (UID: "15171cf4-7349-4232-a057-2d28352faae1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:41.548741 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.546128 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15171cf4-7349-4232-a057-2d28352faae1-kube-api-access-ssdff" pod "15171cf4-7349-4232-a057-2d28352faae1" (UID: "15171cf4-7349-4232-a057-2d28352faae1"). InnerVolumeSpecName "kube-api-access-ssdff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:41.592859 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.591716 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:41.594807 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.593612 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:41.606098 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.606024 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:47:41.639896 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.639861 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15171cf4-7349-4232-a057-2d28352faae1-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:41.640082 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.639899 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15171cf4-7349-4232-a057-2d28352faae1-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:41.640082 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.639916 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssdff\" (UniqueName: \"kubernetes.io/projected/15171cf4-7349-4232-a057-2d28352faae1-kube-api-access-ssdff\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:41.678644 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.678617 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:41.679783 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.679757 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:41.681435 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.681416 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:47:41.913909 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:41.913879 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f"] Oct 30 20:47:41.915249 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:47:41.915224 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25fc72ef_aea6_44a7_9077_40ecd86c99a2.slice/crio-75a55ff8c9aff1e2a5ad7c25f6113d2ce1bafb3e4221a64f704302040cdabc1f WatchSource:0}: Error finding container 75a55ff8c9aff1e2a5ad7c25f6113d2ce1bafb3e4221a64f704302040cdabc1f: Status 404 returned error can't find the container with id 75a55ff8c9aff1e2a5ad7c25f6113d2ce1bafb3e4221a64f704302040cdabc1f Oct 30 20:47:42.061456 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.061424 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d"] Oct 30 20:47:42.066380 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.066334 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-882a72e0-z6q4d"] Oct 30 20:47:42.077405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.077360 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" Oct 30 20:47:42.078014 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.077980 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4" event={"ID":"15171cf4-7349-4232-a057-2d28352faae1","Type":"ContainerDied","Data":"a5d4cb3cbfffbcb29b2e302d2aa76240bda9704a49bf68db30fbe9402e03e4fe"} Oct 30 20:47:42.078132 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.078042 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5d4cb3cbfffbcb29b2e302d2aa76240bda9704a49bf68db30fbe9402e03e4fe" Oct 30 20:47:42.079442 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.079418 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f" event={"ID":"25fc72ef-aea6-44a7-9077-40ecd86c99a2","Type":"ContainerStarted","Data":"75a55ff8c9aff1e2a5ad7c25f6113d2ce1bafb3e4221a64f704302040cdabc1f"} Oct 30 20:47:42.499804 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.499774 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52"] Oct 30 20:47:42.500651 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.500631 5755 cpu_manager.go:414] "containerMap: removing container" podUID="15171cf4-7349-4232-a057-2d28352faae1" containerName="revoke-keys" Oct 30 20:47:42.500718 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.500655 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15171cf4-7349-4232-a057-2d28352faae1" containerName="revoke-keys" Oct 30 20:47:42.500853 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.500840 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="15171cf4-7349-4232-a057-2d28352faae1" containerName="revoke-keys" Oct 30 20:47:42.504883 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.504862 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52" Oct 30 20:47:42.568643 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.566324 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52"] Oct 30 20:47:42.654740 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.654697 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb95c8ca-5mm52\" (UID: \"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52" Oct 30 20:47:42.654937 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.654762 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf7h7\" (UniqueName: \"kubernetes.io/projected/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-kube-api-access-xf7h7\") pod \"maas-api-revoke-sub-fb95c8ca-5mm52\" (UID: \"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52" Oct 30 20:47:42.654937 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.654895 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb95c8ca-5mm52\" (UID: \"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52" Oct 30 20:47:42.749550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.749518 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:42.750492 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.750422 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:42.751319 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.751300 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:47:42.756415 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.756367 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb95c8ca-5mm52\" (UID: \"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52" Oct 30 20:47:42.756533 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.756437 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7h7\" (UniqueName: \"kubernetes.io/projected/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-kube-api-access-xf7h7\") pod \"maas-api-revoke-sub-fb95c8ca-5mm52\" (UID: \"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52" Oct 30 20:47:42.756591 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.756558 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb95c8ca-5mm52\" (UID: \"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52" Oct 30 20:47:42.757292 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.757269 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb95c8ca-5mm52\" (UID: \"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52" Oct 30 20:47:42.763789 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.763762 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb95c8ca-5mm52\" (UID: \"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52" Oct 30 20:47:42.835462 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:42.834431 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7h7\" (UniqueName: \"kubernetes.io/projected/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-kube-api-access-xf7h7\") pod \"maas-api-revoke-sub-fb95c8ca-5mm52\" (UID: \"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52" Oct 30 20:47:43.122896 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:43.122806 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52" Oct 30 20:47:43.194421 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:43.174595 5755 generic.go:356] "Generic (PLEG): container finished" podID="25fc72ef-aea6-44a7-9077-40ecd86c99a2" containerID="44ff3a6825eb266d467996641d7d79df2bdd5daef53f7e637f456025e5635b7c" exitCode=0 Oct 30 20:47:43.196452 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:43.196425 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f" event={"ID":"25fc72ef-aea6-44a7-9077-40ecd86c99a2","Type":"ContainerDied","Data":"44ff3a6825eb266d467996641d7d79df2bdd5daef53f7e637f456025e5635b7c"} Oct 30 20:47:43.309627 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:43.309593 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdba020f-8e2d-40d5-9f78-17970b545e80" path="/var/lib/kubelet/pods/cdba020f-8e2d-40d5-9f78-17970b545e80/volumes" Oct 30 20:47:44.304622 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:44.304581 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:44.351345 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:44.351172 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:44.436414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:44.415485 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:44.483724 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:44.449017 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:44.485342 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:44.458160 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:44.668456 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:44.662952 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52"] Oct 30 20:47:44.686604 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:44.666207 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:44.835855 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:44.834922 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:44.841685 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:44.841649 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Oct 30 20:47:44.869531 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:44.857750 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:44.869866 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:44.869825 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:44.912412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:44.899467 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:44.941410 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:44.940484 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 20:47:45.040614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.040572 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:45.327439 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.309493 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:45.345437 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.329524 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:45.526885 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.503469 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 20:47:45.526885 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.503831 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 20:47:45.526885 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.504099 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 20:47:45.588056 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.588016 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:47:45.588251 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.588092 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:47:45.588251 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.588131 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:47:45.588251 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.588167 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:47:45.915417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.889514 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:45.915417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.889918 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:47:45.915417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.893268 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:47:45.915417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.901605 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:47:45.915417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.902446 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:45.915417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.902843 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:47:45.915417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.903984 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:45.915417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.904265 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:47:45.963036 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.942903 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs"] Oct 30 20:47:45.963036 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.947459 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:45.989995 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.989968 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:47:45.990371 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.990284 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:45.990605 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:45.990543 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:47:46.018968 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.018934 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:46.029486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.029453 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs"] Oct 30 20:47:46.043741 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.043707 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:47:46.045480 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.045451 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:47:46.049201 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.049114 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f" event={"ID":"25fc72ef-aea6-44a7-9077-40ecd86c99a2","Type":"ContainerDied","Data":"75a55ff8c9aff1e2a5ad7c25f6113d2ce1bafb3e4221a64f704302040cdabc1f"} Oct 30 20:47:46.051759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.049328 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75a55ff8c9aff1e2a5ad7c25f6113d2ce1bafb3e4221a64f704302040cdabc1f" Oct 30 20:47:46.051759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.049963 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.051759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.050040 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.051759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.050075 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.051759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.050128 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.051759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.050306 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.051759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.050340 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.051759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.050413 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6svz\" (UniqueName: \"kubernetes.io/projected/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-kube-api-access-l6svz\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.051759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.050442 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.051759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.050497 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.054199 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.054180 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f" Oct 30 20:47:46.055221 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.055161 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52" event={"ID":"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf","Type":"ContainerStarted","Data":"1f53cf385d2ab1a3bd7c27bde1fd6e8c5deb0b9ac79f6b2e05eadfcbcacca57f"} Oct 30 20:47:46.091662 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.091600 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:47:46.279781 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.279748 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25fc72ef-aea6-44a7-9077-40ecd86c99a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25fc72ef-aea6-44a7-9077-40ecd86c99a2-maas-api-openshift-ca\") pod \"25fc72ef-aea6-44a7-9077-40ecd86c99a2\" (UID: \"25fc72ef-aea6-44a7-9077-40ecd86c99a2\") " Oct 30 20:47:46.280965 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.280936 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25fc72ef-aea6-44a7-9077-40ecd86c99a2-kube-api-access-q2f2d\" (UniqueName: \"kubernetes.io/projected/25fc72ef-aea6-44a7-9077-40ecd86c99a2-kube-api-access-q2f2d\") pod \"25fc72ef-aea6-44a7-9077-40ecd86c99a2\" (UID: \"25fc72ef-aea6-44a7-9077-40ecd86c99a2\") " Oct 30 20:47:46.281176 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.281156 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25fc72ef-aea6-44a7-9077-40ecd86c99a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25fc72ef-aea6-44a7-9077-40ecd86c99a2-maas-api-xks-ca\") pod \"25fc72ef-aea6-44a7-9077-40ecd86c99a2\" (UID: \"25fc72ef-aea6-44a7-9077-40ecd86c99a2\") " Oct 30 20:47:46.281643 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.281620 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.281801 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.281783 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.281961 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.281945 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6svz\" (UniqueName: \"kubernetes.io/projected/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-kube-api-access-l6svz\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.282079 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.282064 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.282248 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.282231 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.282941 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.282920 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.283153 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.283133 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.283282 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.283267 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.283429 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.283413 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.284351 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.284268 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.315449 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.315399 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:46.316089 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.316068 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:47:46.331408 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.326443 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:47:46.331408 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.327447 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.331408 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.327950 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.331408 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.328698 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.331408 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.329894 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.331408 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.329957 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:46.335654 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.335627 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25fc72ef-aea6-44a7-9077-40ecd86c99a2-maas-api-openshift-ca" pod "25fc72ef-aea6-44a7-9077-40ecd86c99a2" (UID: "25fc72ef-aea6-44a7-9077-40ecd86c99a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:46.382598 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.379612 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25fc72ef-aea6-44a7-9077-40ecd86c99a2-maas-api-xks-ca" pod "25fc72ef-aea6-44a7-9077-40ecd86c99a2" (UID: "25fc72ef-aea6-44a7-9077-40ecd86c99a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:46.388704 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.387928 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25fc72ef-aea6-44a7-9077-40ecd86c99a2-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:46.388704 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.387966 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25fc72ef-aea6-44a7-9077-40ecd86c99a2-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:46.389302 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.389275 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.516971 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.495849 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.516971 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.509484 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:46.516971 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.509852 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 20:47:46.516971 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.510122 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 20:47:46.571464 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.567240 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.743428 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.741055 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6svz\" (UniqueName: \"kubernetes.io/projected/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-kube-api-access-l6svz\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.938498 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.907476 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:46.965412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.918998 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:46.965412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.940031 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:46.965910 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:46.965869 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 20:47:47.165436 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.026924 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:47.165436 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.058482 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fc72ef-aea6-44a7-9077-40ecd86c99a2-kube-api-access-q2f2d" pod "25fc72ef-aea6-44a7-9077-40ecd86c99a2" (UID: "25fc72ef-aea6-44a7-9077-40ecd86c99a2"). InnerVolumeSpecName "kube-api-access-q2f2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:47.165436 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.145095 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:47:47.165436 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.145986 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:47.256787 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.228048 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2f2d\" (UniqueName: \"kubernetes.io/projected/25fc72ef-aea6-44a7-9077-40ecd86c99a2-kube-api-access-q2f2d\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:47.392428 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.392328 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 20:47:47.512464 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.511296 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 20:47:47.529715 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.529676 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:47:47.877376 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.542927 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:47.877376 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.614703 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 20:47:47.877376 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.786933 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:47.877376 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.876054 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:47.935400 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.935347 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:47:47.935749 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.935734 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:47:47.935890 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.935880 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:47:47.936013 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.936004 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:47:47.976048 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.976011 5755 generic.go:356] "Generic (PLEG): container finished" podID="4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf" containerID="6bc07c3195459306cfb707f6c9835a4524f435e3d66d64530377202c646854e7" exitCode=0 Oct 30 20:47:47.978262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.978238 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f" Oct 30 20:47:47.981521 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.981379 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52" event={"ID":"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf","Type":"ContainerDied","Data":"6bc07c3195459306cfb707f6c9835a4524f435e3d66d64530377202c646854e7"} Oct 30 20:47:47.995455 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.995369 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:47:47.998645 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:47.998628 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:47:48.001025 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:48.001010 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:47:48.001893 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:48.001871 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:47:48.005072 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:48.005058 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:47:48.075866 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:48.075841 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs"] Oct 30 20:47:48.076252 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:47:48.076230 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c005f9c_fdc8_4ae2_bf34_9ffcdf13b044.slice/crio-106f3ccaa3961e65497fdfb071282ab2a20291d6a7b484153d8350e075ccd91c WatchSource:0}: Error finding container 106f3ccaa3961e65497fdfb071282ab2a20291d6a7b484153d8350e075ccd91c: Status 404 returned error can't find the container with id 106f3ccaa3961e65497fdfb071282ab2a20291d6a7b484153d8350e075ccd91c Oct 30 20:47:48.079233 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:48.079199 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:48.079324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:48.079270 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:48.079324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:48.079314 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:47:49.018957 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:49.018917 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" event={"ID":"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044","Type":"ContainerStarted","Data":"e05fff2a11f52f494775123efbc0673cfa31a64f45478d9829faf17fe76a0546"} Oct 30 20:47:49.019463 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:49.018990 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" event={"ID":"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044","Type":"ContainerStarted","Data":"106f3ccaa3961e65497fdfb071282ab2a20291d6a7b484153d8350e075ccd91c"} Oct 30 20:47:49.225878 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:49.225847 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52" Oct 30 20:47:49.313165 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:49.313087 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-maas-api-xks-ca\") pod \"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf\" (UID: \"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf\") " Oct 30 20:47:49.313330 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:49.313187 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-maas-api-openshift-ca\") pod \"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf\" (UID: \"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf\") " Oct 30 20:47:49.313414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:49.313350 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-kube-api-access-xf7h7\" (UniqueName: \"kubernetes.io/projected/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-kube-api-access-xf7h7\") pod \"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf\" (UID: \"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf\") " Oct 30 20:47:49.313573 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:49.313547 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-maas-api-openshift-ca" pod "4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf" (UID: "4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:49.314143 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:49.314119 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:49.315516 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:49.315493 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-maas-api-xks-ca" pod "4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf" (UID: "4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:49.315758 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:49.315737 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-kube-api-access-xf7h7" pod "4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf" (UID: "4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf"). InnerVolumeSpecName "kube-api-access-xf7h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:49.415932 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:49.415902 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf7h7\" (UniqueName: \"kubernetes.io/projected/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-kube-api-access-xf7h7\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:49.415932 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:49.415930 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:49.861905 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:49.861853 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" podStartSLOduration=4.861838764 podStartE2EDuration="4.861838764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:49.857589367 +0000 UTC m=+1232.784146301" watchObservedRunningTime="2026-10-30 20:47:49.861838764 +0000 UTC m=+1232.788395676" Oct 30 20:47:49.966555 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:49.966515 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:49.968210 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:49.968186 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 20:47:49.968324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:49.968232 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" podUID="6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 20:47:50.027059 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:50.027030 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52" Oct 30 20:47:50.027059 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:50.027049 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52" event={"ID":"4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf","Type":"ContainerDied","Data":"1f53cf385d2ab1a3bd7c27bde1fd6e8c5deb0b9ac79f6b2e05eadfcbcacca57f"} Oct 30 20:47:50.027542 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:50.027104 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f53cf385d2ab1a3bd7c27bde1fd6e8c5deb0b9ac79f6b2e05eadfcbcacca57f" Oct 30 20:47:50.967000 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:50.966966 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 20:47:50.967180 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:50.967016 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" podUID="6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 20:47:51.967028 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:51.966996 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 20:47:51.967410 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:51.967050 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" podUID="6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 20:47:52.966168 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:52.966131 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 20:47:52.966586 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:52.966198 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" podUID="6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 20:47:53.169327 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.169298 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq"] Oct 30 20:47:53.200427 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.200353 5755 cpu_manager.go:414] "containerMap: removing container" podUID="4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf" containerName="revoke-keys" Oct 30 20:47:53.200427 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.200405 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf" containerName="revoke-keys" Oct 30 20:47:53.200632 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.200462 5755 cpu_manager.go:414] "containerMap: removing container" podUID="25fc72ef-aea6-44a7-9077-40ecd86c99a2" containerName="revoke-keys" Oct 30 20:47:53.200632 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.200469 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25fc72ef-aea6-44a7-9077-40ecd86c99a2" containerName="revoke-keys" Oct 30 20:47:53.200632 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.200621 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="25fc72ef-aea6-44a7-9077-40ecd86c99a2" containerName="revoke-keys" Oct 30 20:47:53.200811 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.200639 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf" containerName="revoke-keys" Oct 30 20:47:53.204965 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.204932 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq" Oct 30 20:47:53.230600 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.230574 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq"] Oct 30 20:47:53.264127 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.264093 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bggpm\" (UniqueName: \"kubernetes.io/projected/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-kube-api-access-bggpm\") pod \"maas-api-revoke-sub-9d14ed15-9f4qq\" (UID: \"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq" Oct 30 20:47:53.264267 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.264249 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d14ed15-9f4qq\" (UID: \"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq" Oct 30 20:47:53.264462 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.264445 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d14ed15-9f4qq\" (UID: \"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq" Oct 30 20:47:53.280630 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.280604 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:47:53.294529 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.289588 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:47:53.365562 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.365538 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d14ed15-9f4qq\" (UID: \"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq" Oct 30 20:47:53.365690 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.365587 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bggpm\" (UniqueName: \"kubernetes.io/projected/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-kube-api-access-bggpm\") pod \"maas-api-revoke-sub-9d14ed15-9f4qq\" (UID: \"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq" Oct 30 20:47:53.365690 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.365666 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d14ed15-9f4qq\" (UID: \"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq" Oct 30 20:47:53.366451 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.366377 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d14ed15-9f4qq\" (UID: \"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq" Oct 30 20:47:53.368011 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.367982 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d14ed15-9f4qq\" (UID: \"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq" Oct 30 20:47:53.424173 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.424138 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bggpm\" (UniqueName: \"kubernetes.io/projected/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-kube-api-access-bggpm\") pod \"maas-api-revoke-sub-9d14ed15-9f4qq\" (UID: \"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq" Oct 30 20:47:53.520181 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.520105 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq" Oct 30 20:47:53.924457 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.924425 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq"] Oct 30 20:47:53.924949 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:47:53.924910 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16bcfa8_bec1_484d_8cc8_a40ac3e6e4cc.slice/crio-dbc41e648f269a5b7f7c0e75c005f5c9bf21e13f1d2dfbc5aa311905fb1bd500 WatchSource:0}: Error finding container dbc41e648f269a5b7f7c0e75c005f5c9bf21e13f1d2dfbc5aa311905fb1bd500: Status 404 returned error can't find the container with id dbc41e648f269a5b7f7c0e75c005f5c9bf21e13f1d2dfbc5aa311905fb1bd500 Oct 30 20:47:53.967164 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.967141 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 20:47:53.967258 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:53.967185 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" podUID="6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 20:47:54.055335 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:54.055305 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq" event={"ID":"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc","Type":"ContainerStarted","Data":"dbc41e648f269a5b7f7c0e75c005f5c9bf21e13f1d2dfbc5aa311905fb1bd500"} Oct 30 20:47:54.967113 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:54.967070 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 20:47:54.967612 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:54.967126 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" podUID="6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 20:47:55.075314 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:55.075230 5755 generic.go:356] "Generic (PLEG): container finished" podID="f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc" containerID="809412ae3c8d102a1f5953574466bcdf2bd3682d2b8d49e89474fb397a32a9e9" exitCode=0 Oct 30 20:47:55.075314 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:55.075302 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq" event={"ID":"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc","Type":"ContainerDied","Data":"809412ae3c8d102a1f5953574466bcdf2bd3682d2b8d49e89474fb397a32a9e9"} Oct 30 20:47:55.970240 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:55.970213 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:55.970728 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:55.970709 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:55.971427 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:55.971377 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:47:56.231012 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:56.230959 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq" Oct 30 20:47:56.324443 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:56.324419 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-kube-api-access-bggpm\" (UniqueName: \"kubernetes.io/projected/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-kube-api-access-bggpm\") pod \"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc\" (UID: \"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc\") " Oct 30 20:47:56.324601 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:56.324523 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-maas-api-xks-ca\") pod \"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc\" (UID: \"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc\") " Oct 30 20:47:56.324682 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:56.324604 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-maas-api-openshift-ca\") pod \"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc\" (UID: \"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc\") " Oct 30 20:47:56.325026 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:56.324999 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-maas-api-openshift-ca" pod "f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc" (UID: "f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:56.326959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:56.326928 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-maas-api-xks-ca" pod "f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc" (UID: "f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:56.327058 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:56.327020 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-kube-api-access-bggpm" pod "f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc" (UID: "f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc"). InnerVolumeSpecName "kube-api-access-bggpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:56.427240 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:56.427204 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bggpm\" (UniqueName: \"kubernetes.io/projected/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-kube-api-access-bggpm\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:56.427377 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:56.427248 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:56.427377 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:56.427264 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:47:57.091709 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:57.091681 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq" Oct 30 20:47:57.092107 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:57.091720 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq" event={"ID":"f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc","Type":"ContainerDied","Data":"dbc41e648f269a5b7f7c0e75c005f5c9bf21e13f1d2dfbc5aa311905fb1bd500"} Oct 30 20:47:57.092107 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:47:57.091767 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbc41e648f269a5b7f7c0e75c005f5c9bf21e13f1d2dfbc5aa311905fb1bd500" Oct 30 20:48:00.318441 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.318410 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766"] Oct 30 20:48:00.318981 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.318967 5755 cpu_manager.go:414] "containerMap: removing container" podUID="f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc" containerName="revoke-keys" Oct 30 20:48:00.319021 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.318982 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc" containerName="revoke-keys" Oct 30 20:48:00.319093 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.319084 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc" containerName="revoke-keys" Oct 30 20:48:00.322374 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.322358 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.428909 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.428878 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766"] Oct 30 20:48:00.483074 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.483042 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc87293-4490-43f3-851a-05f393007b1a-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.483269 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.483169 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.483269 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.483206 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc87293-4490-43f3-851a-05f393007b1a-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.483406 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.483305 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.483406 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.483371 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.483530 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.483407 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.483530 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.483426 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc87293-4490-43f3-851a-05f393007b1a-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.483530 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.483523 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.483741 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.483562 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7g88\" (UniqueName: \"kubernetes.io/projected/4dc87293-4490-43f3-851a-05f393007b1a-kube-api-access-q7g88\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.584615 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.584535 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.584615 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.584578 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.584615 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.584605 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.584885 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.584734 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc87293-4490-43f3-851a-05f393007b1a-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.584885 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.584805 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.584885 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.584870 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7g88\" (UniqueName: \"kubernetes.io/projected/4dc87293-4490-43f3-851a-05f393007b1a-kube-api-access-q7g88\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.585030 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.584999 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.585084 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.585032 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc87293-4490-43f3-851a-05f393007b1a-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.585174 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.585159 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.585228 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.585170 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.585228 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.585193 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc87293-4490-43f3-851a-05f393007b1a-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.585436 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.585412 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.585647 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.585629 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc87293-4490-43f3-851a-05f393007b1a-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.585704 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.585647 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.587060 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.587037 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.587743 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.587725 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc87293-4490-43f3-851a-05f393007b1a-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.643197 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.643164 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc87293-4490-43f3-851a-05f393007b1a-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.643381 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.643355 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7g88\" (UniqueName: \"kubernetes.io/projected/4dc87293-4490-43f3-851a-05f393007b1a-kube-api-access-q7g88\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:00.936494 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:00.936415 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:01.038605 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.037867 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf"] Oct 30 20:48:01.043000 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.042979 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf" Oct 30 20:48:01.100732 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.100697 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:48:01.100911 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.100828 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf"] Oct 30 20:48:01.100974 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.100908 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:48:01.194215 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.194129 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388b927b-777b-46d2-a43c-fc36dc22a7ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2e8ca55-kgspf\" (UID: \"388b927b-777b-46d2-a43c-fc36dc22a7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf" Oct 30 20:48:01.194419 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.194379 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388b927b-777b-46d2-a43c-fc36dc22a7ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2e8ca55-kgspf\" (UID: \"388b927b-777b-46d2-a43c-fc36dc22a7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf" Oct 30 20:48:01.194506 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.194485 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bdnz\" (UniqueName: \"kubernetes.io/projected/388b927b-777b-46d2-a43c-fc36dc22a7ef-kube-api-access-9bdnz\") pod \"maas-api-revoke-sub-e2e8ca55-kgspf\" (UID: \"388b927b-777b-46d2-a43c-fc36dc22a7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf" Oct 30 20:48:01.296333 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.296301 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388b927b-777b-46d2-a43c-fc36dc22a7ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2e8ca55-kgspf\" (UID: \"388b927b-777b-46d2-a43c-fc36dc22a7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf" Oct 30 20:48:01.296515 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.296456 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388b927b-777b-46d2-a43c-fc36dc22a7ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2e8ca55-kgspf\" (UID: \"388b927b-777b-46d2-a43c-fc36dc22a7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf" Oct 30 20:48:01.296515 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.296498 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bdnz\" (UniqueName: \"kubernetes.io/projected/388b927b-777b-46d2-a43c-fc36dc22a7ef-kube-api-access-9bdnz\") pod \"maas-api-revoke-sub-e2e8ca55-kgspf\" (UID: \"388b927b-777b-46d2-a43c-fc36dc22a7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf" Oct 30 20:48:01.296964 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.296943 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388b927b-777b-46d2-a43c-fc36dc22a7ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2e8ca55-kgspf\" (UID: \"388b927b-777b-46d2-a43c-fc36dc22a7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf" Oct 30 20:48:01.298874 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.298857 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388b927b-777b-46d2-a43c-fc36dc22a7ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2e8ca55-kgspf\" (UID: \"388b927b-777b-46d2-a43c-fc36dc22a7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf" Oct 30 20:48:01.356275 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.356244 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bdnz\" (UniqueName: \"kubernetes.io/projected/388b927b-777b-46d2-a43c-fc36dc22a7ef-kube-api-access-9bdnz\") pod \"maas-api-revoke-sub-e2e8ca55-kgspf\" (UID: \"388b927b-777b-46d2-a43c-fc36dc22a7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf" Oct 30 20:48:01.360130 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.360110 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf" Oct 30 20:48:01.375548 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.375527 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766"] Oct 30 20:48:01.376221 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:48:01.376199 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc87293_4490_43f3_851a_05f393007b1a.slice/crio-30cce890b2fa9850ae93ae9be14dbb3c0ea8db9047a76c43071aa8fcc133584b WatchSource:0}: Error finding container 30cce890b2fa9850ae93ae9be14dbb3c0ea8db9047a76c43071aa8fcc133584b: Status 404 returned error can't find the container with id 30cce890b2fa9850ae93ae9be14dbb3c0ea8db9047a76c43071aa8fcc133584b Oct 30 20:48:01.378476 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.378443 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:01.378549 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.378508 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:01.378549 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.378537 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:01.744011 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:01.743942 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf"] Oct 30 20:48:01.744444 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:48:01.744415 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388b927b_777b_46d2_a43c_fc36dc22a7ef.slice/crio-1a9cf0680ada55837cb7193da4f6cead21986140b564ce9bfdfdd1ced3d866e7 WatchSource:0}: Error finding container 1a9cf0680ada55837cb7193da4f6cead21986140b564ce9bfdfdd1ced3d866e7: Status 404 returned error can't find the container with id 1a9cf0680ada55837cb7193da4f6cead21986140b564ce9bfdfdd1ced3d866e7 Oct 30 20:48:02.128771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:02.128732 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" event={"ID":"4dc87293-4490-43f3-851a-05f393007b1a","Type":"ContainerStarted","Data":"802d17a33c095d0f01e15200d8ff2f62e5a27b46c7c4f2dc6c5c30d6039295a5"} Oct 30 20:48:02.129017 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:02.128797 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" event={"ID":"4dc87293-4490-43f3-851a-05f393007b1a","Type":"ContainerStarted","Data":"30cce890b2fa9850ae93ae9be14dbb3c0ea8db9047a76c43071aa8fcc133584b"} Oct 30 20:48:02.130231 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:02.130203 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf" event={"ID":"388b927b-777b-46d2-a43c-fc36dc22a7ef","Type":"ContainerStarted","Data":"1a9cf0680ada55837cb7193da4f6cead21986140b564ce9bfdfdd1ced3d866e7"} Oct 30 20:48:02.252294 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:02.252171 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" podStartSLOduration=2.252153335 podStartE2EDuration="2.252153335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:02.251797713 +0000 UTC m=+1245.178354651" watchObservedRunningTime="2026-10-30 20:48:02.252153335 +0000 UTC m=+1245.178710252" Oct 30 20:48:02.937498 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:02.937431 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:02.938778 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:02.938751 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:48:02.938922 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:02.938802 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" podUID="4dc87293-4490-43f3-851a-05f393007b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:48:03.137947 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:03.137854 5755 generic.go:356] "Generic (PLEG): container finished" podID="388b927b-777b-46d2-a43c-fc36dc22a7ef" containerID="e45d09601735e14c9a263a097b63c9d023b2ef5d47753a5c9d0518b468864706" exitCode=0 Oct 30 20:48:03.137947 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:03.137933 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf" event={"ID":"388b927b-777b-46d2-a43c-fc36dc22a7ef","Type":"ContainerDied","Data":"e45d09601735e14c9a263a097b63c9d023b2ef5d47753a5c9d0518b468864706"} Oct 30 20:48:03.937017 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:03.936981 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:48:03.937188 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:03.937044 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" podUID="4dc87293-4490-43f3-851a-05f393007b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:48:03.958872 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:03.958843 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8"] Oct 30 20:48:03.963042 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:03.963022 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" Oct 30 20:48:04.021505 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.021475 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8"] Oct 30 20:48:04.036529 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.036505 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6200448-zwtx8\" (UID: \"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" Oct 30 20:48:04.036683 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.036550 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6200448-zwtx8\" (UID: \"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" Oct 30 20:48:04.036749 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.036686 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mhtb\" (UniqueName: \"kubernetes.io/projected/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-kube-api-access-8mhtb\") pod \"maas-api-revoke-sub-c6200448-zwtx8\" (UID: \"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" Oct 30 20:48:04.138465 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.138427 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6200448-zwtx8\" (UID: \"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" Oct 30 20:48:04.138652 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.138489 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6200448-zwtx8\" (UID: \"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" Oct 30 20:48:04.138652 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.138565 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mhtb\" (UniqueName: \"kubernetes.io/projected/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-kube-api-access-8mhtb\") pod \"maas-api-revoke-sub-c6200448-zwtx8\" (UID: \"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" Oct 30 20:48:04.139267 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.139248 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6200448-zwtx8\" (UID: \"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" Oct 30 20:48:04.140984 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.140964 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6200448-zwtx8\" (UID: \"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" Oct 30 20:48:04.199214 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.199142 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mhtb\" (UniqueName: \"kubernetes.io/projected/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-kube-api-access-8mhtb\") pod \"maas-api-revoke-sub-c6200448-zwtx8\" (UID: \"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" Oct 30 20:48:04.303045 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.302628 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" Oct 30 20:48:04.417155 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.416848 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf" Oct 30 20:48:04.443452 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.442659 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388b927b-777b-46d2-a43c-fc36dc22a7ef-kube-api-access-9bdnz\" (UniqueName: \"kubernetes.io/projected/388b927b-777b-46d2-a43c-fc36dc22a7ef-kube-api-access-9bdnz\") pod \"388b927b-777b-46d2-a43c-fc36dc22a7ef\" (UID: \"388b927b-777b-46d2-a43c-fc36dc22a7ef\") " Oct 30 20:48:04.443452 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.442712 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388b927b-777b-46d2-a43c-fc36dc22a7ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388b927b-777b-46d2-a43c-fc36dc22a7ef-maas-api-openshift-ca\") pod \"388b927b-777b-46d2-a43c-fc36dc22a7ef\" (UID: \"388b927b-777b-46d2-a43c-fc36dc22a7ef\") " Oct 30 20:48:04.443452 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.442862 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/388b927b-777b-46d2-a43c-fc36dc22a7ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388b927b-777b-46d2-a43c-fc36dc22a7ef-maas-api-xks-ca\") pod \"388b927b-777b-46d2-a43c-fc36dc22a7ef\" (UID: \"388b927b-777b-46d2-a43c-fc36dc22a7ef\") " Oct 30 20:48:04.444110 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.444081 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388b927b-777b-46d2-a43c-fc36dc22a7ef-maas-api-openshift-ca" pod "388b927b-777b-46d2-a43c-fc36dc22a7ef" (UID: "388b927b-777b-46d2-a43c-fc36dc22a7ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:48:04.444247 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.444223 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388b927b-777b-46d2-a43c-fc36dc22a7ef-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:04.446078 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.446055 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388b927b-777b-46d2-a43c-fc36dc22a7ef-maas-api-xks-ca" pod "388b927b-777b-46d2-a43c-fc36dc22a7ef" (UID: "388b927b-777b-46d2-a43c-fc36dc22a7ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:48:04.446948 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.446921 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388b927b-777b-46d2-a43c-fc36dc22a7ef-kube-api-access-9bdnz" pod "388b927b-777b-46d2-a43c-fc36dc22a7ef" (UID: "388b927b-777b-46d2-a43c-fc36dc22a7ef"). InnerVolumeSpecName "kube-api-access-9bdnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:48:04.546036 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.546002 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bdnz\" (UniqueName: \"kubernetes.io/projected/388b927b-777b-46d2-a43c-fc36dc22a7ef-kube-api-access-9bdnz\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:04.546036 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.546037 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388b927b-777b-46d2-a43c-fc36dc22a7ef-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:04.730432 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.730402 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8"] Oct 30 20:48:04.731079 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:48:04.731058 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3e20f0_ee6c_4544_8bb4_25364ca6b6de.slice/crio-5678430d41a5ae64f5519c6ade8643d70bb349896ab099f3304f1ba5d57844c1 WatchSource:0}: Error finding container 5678430d41a5ae64f5519c6ade8643d70bb349896ab099f3304f1ba5d57844c1: Status 404 returned error can't find the container with id 5678430d41a5ae64f5519c6ade8643d70bb349896ab099f3304f1ba5d57844c1 Oct 30 20:48:04.937187 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.937155 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:48:04.937355 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:04.937217 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" podUID="4dc87293-4490-43f3-851a-05f393007b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:48:05.153869 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:05.153789 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" event={"ID":"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de","Type":"ContainerStarted","Data":"3efbe06a3903c75e46c35a6469aeee0d6590f1159ebbf612a2184b3add133fd0"} Oct 30 20:48:05.154223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:05.153866 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" event={"ID":"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de","Type":"ContainerStarted","Data":"5678430d41a5ae64f5519c6ade8643d70bb349896ab099f3304f1ba5d57844c1"} Oct 30 20:48:05.155270 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:05.155231 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf" Oct 30 20:48:05.155270 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:05.155244 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf" event={"ID":"388b927b-777b-46d2-a43c-fc36dc22a7ef","Type":"ContainerDied","Data":"1a9cf0680ada55837cb7193da4f6cead21986140b564ce9bfdfdd1ced3d866e7"} Oct 30 20:48:05.155446 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:05.155291 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a9cf0680ada55837cb7193da4f6cead21986140b564ce9bfdfdd1ced3d866e7" Oct 30 20:48:05.294687 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:05.291991 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" podStartSLOduration=2.291972924 podStartE2EDuration="2.291972924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:05.269698842 +0000 UTC m=+1248.196255760" watchObservedRunningTime="2026-10-30 20:48:05.291972924 +0000 UTC m=+1248.218529840" Oct 30 20:48:05.937855 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:05.937821 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:48:05.938044 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:05.937876 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" podUID="4dc87293-4490-43f3-851a-05f393007b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:48:06.163893 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:06.163817 5755 generic.go:356] "Generic (PLEG): container finished" podID="cb3e20f0-ee6c-4544-8bb4-25364ca6b6de" containerID="3efbe06a3903c75e46c35a6469aeee0d6590f1159ebbf612a2184b3add133fd0" exitCode=0 Oct 30 20:48:06.164277 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:06.163893 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" event={"ID":"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de","Type":"ContainerDied","Data":"3efbe06a3903c75e46c35a6469aeee0d6590f1159ebbf612a2184b3add133fd0"} Oct 30 20:48:06.937132 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:06.937093 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:48:06.937316 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:06.937169 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" podUID="4dc87293-4490-43f3-851a-05f393007b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:48:07.386232 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:07.386197 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" Oct 30 20:48:07.510902 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:07.510869 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-maas-api-xks-ca\") pod \"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de\" (UID: \"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de\") " Oct 30 20:48:07.511153 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:07.511071 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-kube-api-access-8mhtb\" (UniqueName: \"kubernetes.io/projected/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-kube-api-access-8mhtb\") pod \"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de\" (UID: \"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de\") " Oct 30 20:48:07.511265 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:07.511224 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-maas-api-openshift-ca\") pod \"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de\" (UID: \"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de\") " Oct 30 20:48:07.511820 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:07.511791 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-maas-api-openshift-ca" pod "cb3e20f0-ee6c-4544-8bb4-25364ca6b6de" (UID: "cb3e20f0-ee6c-4544-8bb4-25364ca6b6de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:48:07.523412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:07.523350 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-maas-api-xks-ca" pod "cb3e20f0-ee6c-4544-8bb4-25364ca6b6de" (UID: "cb3e20f0-ee6c-4544-8bb4-25364ca6b6de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:48:07.526590 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:07.526546 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-kube-api-access-8mhtb" pod "cb3e20f0-ee6c-4544-8bb4-25364ca6b6de" (UID: "cb3e20f0-ee6c-4544-8bb4-25364ca6b6de"). InnerVolumeSpecName "kube-api-access-8mhtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:48:07.613798 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:07.613765 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mhtb\" (UniqueName: \"kubernetes.io/projected/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-kube-api-access-8mhtb\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:07.613798 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:07.613800 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:07.613983 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:07.613817 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:07.938791 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:07.938690 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:48:07.938791 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:07.938773 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" podUID="4dc87293-4490-43f3-851a-05f393007b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:48:08.209363 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:08.209276 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" event={"ID":"cb3e20f0-ee6c-4544-8bb4-25364ca6b6de","Type":"ContainerDied","Data":"5678430d41a5ae64f5519c6ade8643d70bb349896ab099f3304f1ba5d57844c1"} Oct 30 20:48:08.209363 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:08.209342 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5678430d41a5ae64f5519c6ade8643d70bb349896ab099f3304f1ba5d57844c1" Oct 30 20:48:08.209363 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:08.209289 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8" Oct 30 20:48:08.943231 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:08.943199 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:08.943839 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:08.943816 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:08.944626 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:08.944604 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:48:11.255622 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:48:11.255586 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:48:11.849972 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:11.849940 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6"] Oct 30 20:48:11.850513 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:11.850499 5755 cpu_manager.go:414] "containerMap: removing container" podUID="cb3e20f0-ee6c-4544-8bb4-25364ca6b6de" containerName="revoke-keys" Oct 30 20:48:11.850565 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:11.850515 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3e20f0-ee6c-4544-8bb4-25364ca6b6de" containerName="revoke-keys" Oct 30 20:48:11.850565 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:11.850539 5755 cpu_manager.go:414] "containerMap: removing container" podUID="388b927b-777b-46d2-a43c-fc36dc22a7ef" containerName="revoke-keys" Oct 30 20:48:11.850565 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:11.850545 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388b927b-777b-46d2-a43c-fc36dc22a7ef" containerName="revoke-keys" Oct 30 20:48:11.850658 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:11.850642 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="388b927b-777b-46d2-a43c-fc36dc22a7ef" containerName="revoke-keys" Oct 30 20:48:11.850658 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:11.850655 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3e20f0-ee6c-4544-8bb4-25364ca6b6de" containerName="revoke-keys" Oct 30 20:48:11.854288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:11.854271 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" Oct 30 20:48:11.915974 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:11.915946 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6"] Oct 30 20:48:11.959215 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:11.959191 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:48:11.959369 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:11.959242 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:48:11.969129 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:11.969095 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db66f48e-1108-4674-b486-568a20c7e7de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3b1c816-lxvr6\" (UID: \"db66f48e-1108-4674-b486-568a20c7e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" Oct 30 20:48:11.969233 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:11.969220 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db66f48e-1108-4674-b486-568a20c7e7de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3b1c816-lxvr6\" (UID: \"db66f48e-1108-4674-b486-568a20c7e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" Oct 30 20:48:11.969296 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:11.969282 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl8bd\" (UniqueName: \"kubernetes.io/projected/db66f48e-1108-4674-b486-568a20c7e7de-kube-api-access-hl8bd\") pod \"maas-api-revoke-sub-c3b1c816-lxvr6\" (UID: \"db66f48e-1108-4674-b486-568a20c7e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" Oct 30 20:48:12.070956 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:12.070930 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db66f48e-1108-4674-b486-568a20c7e7de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3b1c816-lxvr6\" (UID: \"db66f48e-1108-4674-b486-568a20c7e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" Oct 30 20:48:12.071104 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:12.070996 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db66f48e-1108-4674-b486-568a20c7e7de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3b1c816-lxvr6\" (UID: \"db66f48e-1108-4674-b486-568a20c7e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" Oct 30 20:48:12.071104 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:12.071053 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl8bd\" (UniqueName: \"kubernetes.io/projected/db66f48e-1108-4674-b486-568a20c7e7de-kube-api-access-hl8bd\") pod \"maas-api-revoke-sub-c3b1c816-lxvr6\" (UID: \"db66f48e-1108-4674-b486-568a20c7e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" Oct 30 20:48:12.071616 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:12.071593 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db66f48e-1108-4674-b486-568a20c7e7de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3b1c816-lxvr6\" (UID: \"db66f48e-1108-4674-b486-568a20c7e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" Oct 30 20:48:12.073528 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:12.073509 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db66f48e-1108-4674-b486-568a20c7e7de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3b1c816-lxvr6\" (UID: \"db66f48e-1108-4674-b486-568a20c7e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" Oct 30 20:48:12.130736 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:12.130673 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl8bd\" (UniqueName: \"kubernetes.io/projected/db66f48e-1108-4674-b486-568a20c7e7de-kube-api-access-hl8bd\") pod \"maas-api-revoke-sub-c3b1c816-lxvr6\" (UID: \"db66f48e-1108-4674-b486-568a20c7e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" Oct 30 20:48:12.167332 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:12.167310 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" Oct 30 20:48:12.551341 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:12.551312 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6"] Oct 30 20:48:12.551778 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:48:12.551464 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb66f48e_1108_4674_b486_568a20c7e7de.slice/crio-3cc5c1baf736b92bc2f7ca85886e630be940610c429da940e08e6f689e6cc990 WatchSource:0}: Error finding container 3cc5c1baf736b92bc2f7ca85886e630be940610c429da940e08e6f689e6cc990: Status 404 returned error can't find the container with id 3cc5c1baf736b92bc2f7ca85886e630be940610c429da940e08e6f689e6cc990 Oct 30 20:48:13.253030 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:13.253001 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" event={"ID":"db66f48e-1108-4674-b486-568a20c7e7de","Type":"ContainerStarted","Data":"ff154c042bdfb047ac2a11e6fd4f8292d13162524a7f82cb70440cad05733942"} Oct 30 20:48:13.253225 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:13.253049 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" event={"ID":"db66f48e-1108-4674-b486-568a20c7e7de","Type":"ContainerStarted","Data":"3cc5c1baf736b92bc2f7ca85886e630be940610c429da940e08e6f689e6cc990"} Oct 30 20:48:13.374143 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:13.374096 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" podStartSLOduration=2.374081505 podStartE2EDuration="2.374081505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:13.37146189 +0000 UTC m=+1256.298018820" watchObservedRunningTime="2026-10-30 20:48:13.374081505 +0000 UTC m=+1256.300638420" Oct 30 20:48:14.260758 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:14.260718 5755 generic.go:356] "Generic (PLEG): container finished" podID="db66f48e-1108-4674-b486-568a20c7e7de" containerID="ff154c042bdfb047ac2a11e6fd4f8292d13162524a7f82cb70440cad05733942" exitCode=0 Oct 30 20:48:14.261109 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:14.260807 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" event={"ID":"db66f48e-1108-4674-b486-568a20c7e7de","Type":"ContainerDied","Data":"ff154c042bdfb047ac2a11e6fd4f8292d13162524a7f82cb70440cad05733942"} Oct 30 20:48:15.416104 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.416080 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" Oct 30 20:48:15.516142 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.516111 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db66f48e-1108-4674-b486-568a20c7e7de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db66f48e-1108-4674-b486-568a20c7e7de-maas-api-openshift-ca\") pod \"db66f48e-1108-4674-b486-568a20c7e7de\" (UID: \"db66f48e-1108-4674-b486-568a20c7e7de\") " Oct 30 20:48:15.516321 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.516155 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db66f48e-1108-4674-b486-568a20c7e7de-kube-api-access-hl8bd\" (UniqueName: \"kubernetes.io/projected/db66f48e-1108-4674-b486-568a20c7e7de-kube-api-access-hl8bd\") pod \"db66f48e-1108-4674-b486-568a20c7e7de\" (UID: \"db66f48e-1108-4674-b486-568a20c7e7de\") " Oct 30 20:48:15.516321 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.516264 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db66f48e-1108-4674-b486-568a20c7e7de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db66f48e-1108-4674-b486-568a20c7e7de-maas-api-xks-ca\") pod \"db66f48e-1108-4674-b486-568a20c7e7de\" (UID: \"db66f48e-1108-4674-b486-568a20c7e7de\") " Oct 30 20:48:15.516555 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.516531 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db66f48e-1108-4674-b486-568a20c7e7de-maas-api-openshift-ca" pod "db66f48e-1108-4674-b486-568a20c7e7de" (UID: "db66f48e-1108-4674-b486-568a20c7e7de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:48:15.516786 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.516771 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db66f48e-1108-4674-b486-568a20c7e7de-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:15.518425 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.518345 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db66f48e-1108-4674-b486-568a20c7e7de-maas-api-xks-ca" pod "db66f48e-1108-4674-b486-568a20c7e7de" (UID: "db66f48e-1108-4674-b486-568a20c7e7de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:48:15.518521 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.518425 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db66f48e-1108-4674-b486-568a20c7e7de-kube-api-access-hl8bd" pod "db66f48e-1108-4674-b486-568a20c7e7de" (UID: "db66f48e-1108-4674-b486-568a20c7e7de"). InnerVolumeSpecName "kube-api-access-hl8bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:48:15.617952 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.617917 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db66f48e-1108-4674-b486-568a20c7e7de-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:15.617952 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.617946 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl8bd\" (UniqueName: \"kubernetes.io/projected/db66f48e-1108-4674-b486-568a20c7e7de-kube-api-access-hl8bd\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:15.681224 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.681188 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv"] Oct 30 20:48:15.681768 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.681753 5755 cpu_manager.go:414] "containerMap: removing container" podUID="db66f48e-1108-4674-b486-568a20c7e7de" containerName="revoke-keys" Oct 30 20:48:15.681824 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.681770 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db66f48e-1108-4674-b486-568a20c7e7de" containerName="revoke-keys" Oct 30 20:48:15.681880 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.681871 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="db66f48e-1108-4674-b486-568a20c7e7de" containerName="revoke-keys" Oct 30 20:48:15.685511 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.685489 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.746879 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.746851 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv"] Oct 30 20:48:15.821556 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.821472 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.821757 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.821736 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.821822 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.821769 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.821874 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.821860 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gzkw\" (UniqueName: \"kubernetes.io/projected/fe01d860-8f2e-4a07-9e35-d25f0fa39302-kube-api-access-6gzkw\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.821941 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.821928 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.822034 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.822011 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.822124 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.822100 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.822225 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.822210 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.822295 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.822279 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.924043 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.924012 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.924220 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.924071 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.924220 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.924143 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.924220 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.924203 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.924372 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.924229 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.924372 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.924264 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gzkw\" (UniqueName: \"kubernetes.io/projected/fe01d860-8f2e-4a07-9e35-d25f0fa39302-kube-api-access-6gzkw\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.924372 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.924363 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.924575 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.924428 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.924575 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.924476 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.924575 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.924522 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.924733 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.924572 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.924733 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.924606 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.924953 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.924930 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.925082 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.925062 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.926737 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.926717 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.926995 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.926976 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.983247 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.983211 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.983349 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.983284 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gzkw\" (UniqueName: \"kubernetes.io/projected/fe01d860-8f2e-4a07-9e35-d25f0fa39302-kube-api-access-6gzkw\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:15.998909 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:15.998884 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:16.275586 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:16.275551 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" Oct 30 20:48:16.275586 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:16.275564 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6" event={"ID":"db66f48e-1108-4674-b486-568a20c7e7de","Type":"ContainerDied","Data":"3cc5c1baf736b92bc2f7ca85886e630be940610c429da940e08e6f689e6cc990"} Oct 30 20:48:16.275788 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:16.275609 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cc5c1baf736b92bc2f7ca85886e630be940610c429da940e08e6f689e6cc990" Oct 30 20:48:16.435747 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:48:16.435722 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe01d860_8f2e_4a07_9e35_d25f0fa39302.slice/crio-a4b6f4618bad645cc050154f66fdb7825d9d2d09b2045b95b6af2ffba450c3a7 WatchSource:0}: Error finding container a4b6f4618bad645cc050154f66fdb7825d9d2d09b2045b95b6af2ffba450c3a7: Status 404 returned error can't find the container with id a4b6f4618bad645cc050154f66fdb7825d9d2d09b2045b95b6af2ffba450c3a7 Oct 30 20:48:16.438112 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:16.438077 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:16.438233 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:16.438157 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:16.438233 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:16.438204 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:16.440252 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:16.440231 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv"] Oct 30 20:48:17.297572 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:17.297531 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" event={"ID":"fe01d860-8f2e-4a07-9e35-d25f0fa39302","Type":"ContainerStarted","Data":"e5565c5246cacd2f5b8588dfb66a8e87fd57a8717835c7286e7528f719e25889"} Oct 30 20:48:17.297898 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:17.297878 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" event={"ID":"fe01d860-8f2e-4a07-9e35-d25f0fa39302","Type":"ContainerStarted","Data":"a4b6f4618bad645cc050154f66fdb7825d9d2d09b2045b95b6af2ffba450c3a7"} Oct 30 20:48:17.468051 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:17.468020 5755 scope.go:122] "RemoveContainer" containerID="b0a9631a3017254568f68669cf360f135322163feab8936aeb862b5e7f5bd420" Oct 30 20:48:17.542410 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:17.536586 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" podStartSLOduration=2.536552141 podStartE2EDuration="2.536552141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:17.468930634 +0000 UTC m=+1260.395487586" watchObservedRunningTime="2026-10-30 20:48:17.536552141 +0000 UTC m=+1260.463109056" Oct 30 20:48:18.004744 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:18.004712 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:18.016651 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:18.016605 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:48:18.016940 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:18.016908 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:48:19.003411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:19.002847 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:48:19.003411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:19.002929 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:48:19.443467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:19.443212 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:19.522424 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:19.521480 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 20:48:19.878540 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:19.878489 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:48:19.879229 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:19.879195 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 20:48:19.933535 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:19.932663 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 20:48:19.939503 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:19.939462 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:20.000203 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.000160 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:48:20.000542 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.000485 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:48:20.004525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.004465 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:20.011527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.011471 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:20.068531 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.068483 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:20.097570 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.097519 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:20.150666 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.150537 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:20.329412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.328494 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:20.537412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.490750 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:20.537412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.522182 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:20.575412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.538163 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:48:20.915545 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.901204 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:20.932805 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.928754 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:48:20.953412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.938492 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:20.953412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.939655 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:48:20.953412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.940487 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 20:48:20.953412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.952664 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:48:20.954516 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.953961 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:20.982465 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.956595 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Oct 30 20:48:20.998419 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:20.987478 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:21.024866 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:21.013485 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:21.036444 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:21.015372 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:48:21.036719 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:21.036689 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:48:21.036922 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:21.036904 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:48:21.059171 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:21.055160 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:21.059951 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:21.059929 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:48:21.071419 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:21.069311 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:21.071419 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:21.070533 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:48:21.139439 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:21.106468 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:48:21.161918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:21.157513 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 20:48:21.162435 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:21.162411 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:48:21.162639 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:21.162624 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:48:21.376414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:21.351676 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:21.560244 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:21.560210 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:48:21.584531 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:21.584490 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:21.909300 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:21.908510 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:22.105412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.091809 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:22.105412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.092513 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:48:22.115053 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.114995 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:22.117500 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.117468 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 20:48:22.118532 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.118509 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:48:22.118772 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.118745 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:48:22.137760 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.122624 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:22.139285 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.124958 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:22.140150 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.140127 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:48:22.143847 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.125000 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:48:22.200470 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.190252 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 20:48:22.268301 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.268240 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:48:22.274614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.271510 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 20:48:22.331792 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.331715 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:48:22.332903 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.332862 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" podUID="1e3d6326-166f-47f0-888e-f7a5e8464aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:22.547410 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.543885 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 20:48:22.570871 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.570821 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:22.573206 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.573173 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:22.589563 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.589504 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 20:48:22.961476 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:22.958503 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 20:48:23.099418 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.099113 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:23.119758 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.119117 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:23.119758 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.119493 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 20:48:23.120427 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.120354 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:23.139773 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.139740 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:48:23.140019 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.139991 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:48:23.141028 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.141000 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:23.141527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.141178 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:23.153475 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.153428 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:23.197871 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.197814 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:23.296414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.282897 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:48:23.298661 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.298620 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:23.414413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.411615 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:48:23.414413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.411681 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:48:23.609533 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.579272 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:23.622421 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.610469 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:23.752965 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.752935 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:48:23.993637 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.993497 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:48:23.993637 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:23.993563 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" podUID="56c12235-feb4-43a4-a2e3-32eb61584f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:24.014950 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.014823 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:48:24.014950 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.014883 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:48:24.161420 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.147590 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:24.228441 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.221582 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 20:48:24.228441 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.222004 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 20:48:24.233445 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.229318 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:48:24.238484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.237495 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:48:24.238484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.237549 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" podUID="e41e4f75-ea02-41f4-9698-c064cefa94ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:24.239770 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.239442 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:24.248475 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.240437 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:24.293651 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.287537 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:24.293651 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.293512 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:48:24.293651 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.293568 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" podUID="89ccbe51-66d9-4fab-836f-50eaff323f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:24.323488 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.313575 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 20:48:24.399269 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.397502 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:24.424858 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.419490 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:48:24.424858 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.419565 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" podUID="ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:24.581061 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.580495 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 20:48:24.581061 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.580896 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:48:24.581061 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.580941 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" podUID="9847155d-936e-482a-a172-d79a44533b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:24.646652 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.634607 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 30 20:48:24.696563 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.696518 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:24.712166 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.712122 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:24.959857 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.959754 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 20:48:24.992117 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.992082 5755 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:48:24.992788 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.992756 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" podUID="4dc87293-4490-43f3-851a-05f393007b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:24.999495 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.999368 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:48:24.999495 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:24.999431 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:48:25.153215 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.153176 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:25.232550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.232504 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 20:48:25.255648 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.250840 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 20:48:25.291495 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.250776 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:25.291495 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.250920 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:48:25.291495 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.275941 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:25.291495 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.287077 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 20:48:25.433417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.395714 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 20:48:25.433417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.402741 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:25.444634 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.444603 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:48:25.613573 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.590066 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:25.714277 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.711755 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:48:25.734571 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.734526 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:25.824150 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.824122 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:48:25.911534 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.910670 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:48:25.913999 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.913072 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:48:25.916688 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.916538 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:48:25.919659 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.919638 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:48:25.923507 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.923484 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:48:25.923601 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.923568 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:48:25.924573 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.924556 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:48:25.926159 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:25.926141 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:48:26.008415 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:26.007759 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:48:26.008415 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:26.007824 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:48:27.000902 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:27.000869 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:48:27.001532 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:27.001480 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:48:28.000135 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:28.000098 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:48:28.000297 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:28.000152 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:48:28.999460 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:28.999420 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:48:28.999957 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:28.999474 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:48:30.947249 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:30.947216 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:30.948229 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:30.948201 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:30.949527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:30.949502 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 20:48:41.264461 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:48:41.264415 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:48:45.940439 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:45.940373 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56"] Oct 30 20:48:45.945758 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:45.945737 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.046072 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.046038 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.046072 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.046074 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.046344 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.046105 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.046344 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.046174 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.046523 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.046339 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135653bb-b684-408c-9ee5-7a5930f3ff1d-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.046523 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.046373 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.046523 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.046506 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2xl5\" (UniqueName: \"kubernetes.io/projected/135653bb-b684-408c-9ee5-7a5930f3ff1d-kube-api-access-h2xl5\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.046704 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.046539 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.046704 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.046607 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.051750 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.051718 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56"] Oct 30 20:48:46.183954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.179714 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2xl5\" (UniqueName: \"kubernetes.io/projected/135653bb-b684-408c-9ee5-7a5930f3ff1d-kube-api-access-h2xl5\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.183954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.179782 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.183954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.180134 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.183954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.180208 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.183954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.180226 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.183954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.180254 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.183954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.180412 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.183954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.180452 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.183954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.180602 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135653bb-b684-408c-9ee5-7a5930f3ff1d-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.183954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.180645 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.183954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.180997 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.183954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.181237 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.183954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.181750 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.183954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.182779 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135653bb-b684-408c-9ee5-7a5930f3ff1d-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.207757 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.207698 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.209909 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.209865 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.242181 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.242147 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2xl5\" (UniqueName: \"kubernetes.io/projected/135653bb-b684-408c-9ee5-7a5930f3ff1d-kube-api-access-h2xl5\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.245769 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.245742 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.261815 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.261789 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:46.415260 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.415230 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6"] Oct 30 20:48:46.421070 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.421046 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.484170 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.484132 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.484423 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.484259 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.484423 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.484350 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.484570 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.484459 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qqzv\" (UniqueName: \"kubernetes.io/projected/dec4885f-2622-40b5-bc12-e15f5ec1e51c-kube-api-access-5qqzv\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.484705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.484566 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.484814 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.484793 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.484955 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.484936 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.485034 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.484979 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.485104 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.485084 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6"] Oct 30 20:48:46.485176 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.485087 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.586275 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.586239 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.586598 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.586325 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.586598 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.586503 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.586598 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.586530 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.586598 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.586563 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.586994 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.586660 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qqzv\" (UniqueName: \"kubernetes.io/projected/dec4885f-2622-40b5-bc12-e15f5ec1e51c-kube-api-access-5qqzv\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.586994 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.586701 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.586994 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.586832 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.586994 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.586913 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.587290 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.587122 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.588094 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.588071 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.588171 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.588086 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.588593 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.588565 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.588837 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.588788 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.590973 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.590942 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.591533 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.591509 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.647966 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.647928 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.648299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.648275 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qqzv\" (UniqueName: \"kubernetes.io/projected/dec4885f-2622-40b5-bc12-e15f5ec1e51c-kube-api-access-5qqzv\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:46.713196 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.713163 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56"] Oct 30 20:48:46.713505 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:48:46.713477 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135653bb_b684_408c_9ee5_7a5930f3ff1d.slice/crio-43771bebee63ac04de648ff6929d0c6839a7c594ed0143a9cec6a5e79ad306a8 WatchSource:0}: Error finding container 43771bebee63ac04de648ff6929d0c6839a7c594ed0143a9cec6a5e79ad306a8: Status 404 returned error can't find the container with id 43771bebee63ac04de648ff6929d0c6839a7c594ed0143a9cec6a5e79ad306a8 Oct 30 20:48:46.717149 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.717112 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:46.717240 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.717193 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:46.717240 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.717224 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:46.739873 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:46.739819 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:47.201974 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:48:47.201944 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec4885f_2622_40b5_bc12_e15f5ec1e51c.slice/crio-da8b6837d1f7f49b59694ef2148f8976ca4d26e8083df32e469c5e436c509073 WatchSource:0}: Error finding container da8b6837d1f7f49b59694ef2148f8976ca4d26e8083df32e469c5e436c509073: Status 404 returned error can't find the container with id da8b6837d1f7f49b59694ef2148f8976ca4d26e8083df32e469c5e436c509073 Oct 30 20:48:47.202293 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:47.202193 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6"] Oct 30 20:48:47.206313 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:47.205272 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:47.206313 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:47.205349 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:47.206313 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:47.205403 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:47.229546 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:47.229514 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" event={"ID":"135653bb-b684-408c-9ee5-7a5930f3ff1d","Type":"ContainerStarted","Data":"0c81c3796b1d11b3104466973abaff770a700cf2dcdb7ebf2f56a9a2baf06fe0"} Oct 30 20:48:47.229663 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:47.229576 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" event={"ID":"135653bb-b684-408c-9ee5-7a5930f3ff1d","Type":"ContainerStarted","Data":"43771bebee63ac04de648ff6929d0c6839a7c594ed0143a9cec6a5e79ad306a8"} Oct 30 20:48:47.231996 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:47.231962 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" event={"ID":"dec4885f-2622-40b5-bc12-e15f5ec1e51c","Type":"ContainerStarted","Data":"da8b6837d1f7f49b59694ef2148f8976ca4d26e8083df32e469c5e436c509073"} Oct 30 20:48:47.262647 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:47.262616 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:47.264170 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:47.264147 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:48:47.264317 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:47.264192 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" podUID="135653bb-b684-408c-9ee5-7a5930f3ff1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 20:48:47.352351 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:47.352294 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" podStartSLOduration=2.352278086 podStartE2EDuration="2.352278086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:47.350462999 +0000 UTC m=+1290.277019928" watchObservedRunningTime="2026-10-30 20:48:47.352278086 +0000 UTC m=+1290.278835001" Oct 30 20:48:48.244090 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:48.244030 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" event={"ID":"dec4885f-2622-40b5-bc12-e15f5ec1e51c","Type":"ContainerStarted","Data":"3a257893ff509f695560ca9a3dbd1820eb46808f2f3ae64ab1dadb8bf6e9fc40"} Oct 30 20:48:48.262962 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:48.262924 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:48:48.263131 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:48.262972 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" podUID="135653bb-b684-408c-9ee5-7a5930f3ff1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 20:48:48.506471 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:48.500717 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" podStartSLOduration=2.500694834 podStartE2EDuration="2.500694834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:48.42200979 +0000 UTC m=+1291.348566707" watchObservedRunningTime="2026-10-30 20:48:48.500694834 +0000 UTC m=+1291.427251752" Oct 30 20:48:48.741488 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:48.741449 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:48.743954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:48.743927 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:48:48.744105 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:48.743992 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" podUID="dec4885f-2622-40b5-bc12-e15f5ec1e51c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 20:48:49.262986 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:49.262958 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:48:49.263374 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:49.263019 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" podUID="135653bb-b684-408c-9ee5-7a5930f3ff1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 20:48:49.741180 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:49.741145 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:48:49.741363 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:49.741194 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" podUID="dec4885f-2622-40b5-bc12-e15f5ec1e51c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 20:48:50.262312 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:50.262282 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:48:50.262501 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:50.262350 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" podUID="135653bb-b684-408c-9ee5-7a5930f3ff1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 20:48:50.740778 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:50.740744 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:48:50.741177 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:50.740793 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" podUID="dec4885f-2622-40b5-bc12-e15f5ec1e51c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 20:48:51.262869 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:51.262839 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:48:51.263038 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:51.262902 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" podUID="135653bb-b684-408c-9ee5-7a5930f3ff1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 20:48:51.740919 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:51.740888 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:48:51.741426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:51.740946 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" podUID="dec4885f-2622-40b5-bc12-e15f5ec1e51c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 20:48:52.262615 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:52.262586 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:48:52.262843 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:52.262821 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" podUID="135653bb-b684-408c-9ee5-7a5930f3ff1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 20:48:52.740850 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:52.740815 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:48:52.741024 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:52.740863 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" podUID="dec4885f-2622-40b5-bc12-e15f5ec1e51c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 20:48:53.266711 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:53.265818 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:48:53.266711 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:53.265878 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" podUID="135653bb-b684-408c-9ee5-7a5930f3ff1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 20:48:53.740735 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:53.740700 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:48:53.740905 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:53.740781 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" podUID="dec4885f-2622-40b5-bc12-e15f5ec1e51c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 20:48:54.268120 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:54.268092 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:54.268606 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:54.268586 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:54.269273 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:54.269251 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:48:54.821534 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:54.821502 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:54.823276 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:54.822143 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:48:54.823276 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:48:54.823254 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:49:05.302426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:05.298468 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 20:49:05.578469 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:05.577476 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:05.600411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:05.594464 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:05.924458 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:05.923933 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 20:49:06.041642 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:06.041581 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:06.131200 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:06.131142 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 20:49:06.153440 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:06.153344 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:06.187518 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:06.154014 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:49:06.187518 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:06.154349 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:49:06.262450 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:06.251815 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:06.278872 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:06.277480 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:49:06.279336 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:06.279300 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:06.344412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:06.343589 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:06.344412 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:06.344225 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:49:06.637440 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:06.610850 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:06.718558 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:06.666503 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:49:06.718558 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:06.614724 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:06.718558 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:06.667633 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:49:06.952469 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:06.938435 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:06.952469 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:06.951007 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:49:07.071413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.071226 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:07.103420 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.072845 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:49:07.241187 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.131585 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:07.241187 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.146734 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:49:07.241187 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.155465 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:07.241187 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.157167 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:49:07.241187 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.205504 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:07.241187 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.206493 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:49:07.303788 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.301517 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:07.303788 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.302231 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 20:49:07.331871 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.331843 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:49:07.332121 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.332109 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:49:07.336411 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.334467 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 20:49:07.336546 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.336414 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:49:07.339478 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.339454 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:49:07.364739 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.362502 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:07.364739 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.362818 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:49:07.370779 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.370712 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:49:07.397918 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.397890 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 20:49:07.428433 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.425543 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 20:49:07.452360 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.452331 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:49:07.482893 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.482868 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 20:49:07.502641 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.502543 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:49:07.503012 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.502992 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:49:07.503611 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:07.503570 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:49:11.254875 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:49:11.254839 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:49:17.640583 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:17.640551 5755 scope.go:122] "RemoveContainer" containerID="a62fa543769404233279205da405f8c7fd0b7e6a49b143113f05e2559b94f2ea" Oct 30 20:49:17.655655 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:17.655626 5755 scope.go:122] "RemoveContainer" containerID="3e25f9820ad45a897100bb3d3db3ca7afcc17cc3be3dc73dc7cf3090434d4f5a" Oct 30 20:49:17.665163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:17.665141 5755 scope.go:122] "RemoveContainer" containerID="20cfb2f6ce1752dc13033700df7c96e3e173d0efc70af11d9175dcdbae3d232e" Oct 30 20:49:18.433487 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:18.433447 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s"] Oct 30 20:49:18.439183 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:18.439161 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:18.465970 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:18.465946 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a0ee40-e313-40a2-9f98-865817e6f0cd-openshift-service-ca\") pod \"maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:18.466093 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:18.466001 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-maas-api-tls\") pod \"maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:18.466167 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:18.466155 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-metrics-certs\") pod \"maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:18.466207 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:18.466176 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqbwq\" (UniqueName: \"kubernetes.io/projected/d0a0ee40-e313-40a2-9f98-865817e6f0cd-kube-api-access-dqbwq\") pod \"maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:18.493603 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:18.493582 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-dc0e9f64-serving-cert\"" Oct 30 20:49:18.547139 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:18.547099 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s"] Oct 30 20:49:18.567573 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:18.567541 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-metrics-certs\") pod \"maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:18.567738 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:18.567584 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqbwq\" (UniqueName: \"kubernetes.io/projected/d0a0ee40-e313-40a2-9f98-865817e6f0cd-kube-api-access-dqbwq\") pod \"maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:18.567819 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:18.567799 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a0ee40-e313-40a2-9f98-865817e6f0cd-openshift-service-ca\") pod \"maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:18.567888 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:18.567873 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-maas-api-tls\") pod \"maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:18.568048 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:49:18.568033 5755 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-serving-cert: secret "maas-api-e2e-migrate-dc0e9f64-serving-cert" not found Oct 30 20:49:18.568131 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:49:18.568110 5755 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-maas-api-tls podName:d0a0ee40-e313-40a2-9f98-865817e6f0cd nodeName:}" failed. No retries permitted until 2026-10-30 20:49:19.068089045 +0000 UTC m=+1321.994645940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-maas-api-tls") pod "maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" (UID: "d0a0ee40-e313-40a2-9f98-865817e6f0cd") : secret "maas-api-e2e-migrate-dc0e9f64-serving-cert" not found Oct 30 20:49:18.568564 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:18.568543 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a0ee40-e313-40a2-9f98-865817e6f0cd-openshift-service-ca\") pod \"maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:18.570689 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:18.570669 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-metrics-certs\") pod \"maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:18.628584 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:18.628558 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqbwq\" (UniqueName: \"kubernetes.io/projected/d0a0ee40-e313-40a2-9f98-865817e6f0cd-kube-api-access-dqbwq\") pod \"maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:19.075737 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:19.075705 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-maas-api-tls\") pod \"maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:19.078276 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:19.078257 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-maas-api-tls\") pod \"maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:19.351796 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:19.351716 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:19.733712 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:49:19.733674 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a0ee40_e313_40a2_9f98_865817e6f0cd.slice/crio-9fcf37b4ed66bfcb3d0cbe3ea4e39e66753e1e1a7d6be642cec1fd0ad2447498 WatchSource:0}: Error finding container 9fcf37b4ed66bfcb3d0cbe3ea4e39e66753e1e1a7d6be642cec1fd0ad2447498: Status 404 returned error can't find the container with id 9fcf37b4ed66bfcb3d0cbe3ea4e39e66753e1e1a7d6be642cec1fd0ad2447498 Oct 30 20:49:19.734803 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:19.734776 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s"] Oct 30 20:49:19.735218 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:19.735202 5755 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:49:19.960713 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:19.960683 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" event={"ID":"d0a0ee40-e313-40a2-9f98-865817e6f0cd","Type":"ContainerStarted","Data":"9fcf37b4ed66bfcb3d0cbe3ea4e39e66753e1e1a7d6be642cec1fd0ad2447498"} Oct 30 20:49:20.969692 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:20.969660 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" event={"ID":"d0a0ee40-e313-40a2-9f98-865817e6f0cd","Type":"ContainerStarted","Data":"85107efaf22f6266a7a2fbb30f4345c48e9ad7ac6b3cd7c80674f9f992398d29"} Oct 30 20:49:20.970099 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:20.969780 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:21.090503 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:21.090439 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" podStartSLOduration=2.218268121 podStartE2EDuration="3.090420833s" totalImagesPullingTime="872.152712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:21.088355498 +0000 UTC m=+1324.014912414" watchObservedRunningTime="2026-10-30 20:49:21.090420833 +0000 UTC m=+1324.016977756" Oct 30 20:49:26.982688 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:26.982658 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:49:28.393536 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:28.393502 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn"] Oct 30 20:49:28.397932 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:28.397915 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:28.495222 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:28.495183 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-metrics-certs\") pod \"maas-api-e2e-ait-f9782c31-f96989bd8-tskqn\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:28.495431 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:28.495247 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbx9x\" (UniqueName: \"kubernetes.io/projected/209affb4-e050-418c-b522-d120e03a8f0c-kube-api-access-dbx9x\") pod \"maas-api-e2e-ait-f9782c31-f96989bd8-tskqn\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:28.495579 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:28.495557 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209affb4-e050-418c-b522-d120e03a8f0c-openshift-service-ca\") pod \"maas-api-e2e-ait-f9782c31-f96989bd8-tskqn\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:28.495655 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:28.495592 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-maas-api-tls\") pod \"maas-api-e2e-ait-f9782c31-f96989bd8-tskqn\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:28.501726 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:28.501691 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f9782c31-serving-cert\"" Oct 30 20:49:28.506815 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:28.506794 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn"] Oct 30 20:49:28.596784 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:28.596751 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbx9x\" (UniqueName: \"kubernetes.io/projected/209affb4-e050-418c-b522-d120e03a8f0c-kube-api-access-dbx9x\") pod \"maas-api-e2e-ait-f9782c31-f96989bd8-tskqn\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:28.597052 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:28.597030 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209affb4-e050-418c-b522-d120e03a8f0c-openshift-service-ca\") pod \"maas-api-e2e-ait-f9782c31-f96989bd8-tskqn\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:28.597152 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:28.597066 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-maas-api-tls\") pod \"maas-api-e2e-ait-f9782c31-f96989bd8-tskqn\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:28.597225 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:28.597210 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-metrics-certs\") pod \"maas-api-e2e-ait-f9782c31-f96989bd8-tskqn\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:28.597303 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:49:28.597216 5755 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-serving-cert: secret "maas-api-e2e-ait-f9782c31-serving-cert" not found Oct 30 20:49:28.597369 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:49:28.597359 5755 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-maas-api-tls podName:209affb4-e050-418c-b522-d120e03a8f0c nodeName:}" failed. No retries permitted until 2026-10-30 20:49:29.097342039 +0000 UTC m=+1332.023899175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-maas-api-tls") pod "maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" (UID: "209affb4-e050-418c-b522-d120e03a8f0c") : secret "maas-api-e2e-ait-f9782c31-serving-cert" not found Oct 30 20:49:28.597702 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:28.597682 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209affb4-e050-418c-b522-d120e03a8f0c-openshift-service-ca\") pod \"maas-api-e2e-ait-f9782c31-f96989bd8-tskqn\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:28.599678 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:28.599664 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-metrics-certs\") pod \"maas-api-e2e-ait-f9782c31-f96989bd8-tskqn\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:28.661080 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:28.660975 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbx9x\" (UniqueName: \"kubernetes.io/projected/209affb4-e050-418c-b522-d120e03a8f0c-kube-api-access-dbx9x\") pod \"maas-api-e2e-ait-f9782c31-f96989bd8-tskqn\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:29.103856 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:29.103815 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-maas-api-tls\") pod \"maas-api-e2e-ait-f9782c31-f96989bd8-tskqn\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:29.106400 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:29.106358 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-maas-api-tls\") pod \"maas-api-e2e-ait-f9782c31-f96989bd8-tskqn\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:29.311424 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:29.311378 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:29.691763 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:29.691732 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn"] Oct 30 20:49:29.692103 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:49:29.691819 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209affb4_e050_418c_b522_d120e03a8f0c.slice/crio-07f236f0d771b1916b27fb5d0810385ea3efdc051768b75ed99a6dbe98b7a3cd WatchSource:0}: Error finding container 07f236f0d771b1916b27fb5d0810385ea3efdc051768b75ed99a6dbe98b7a3cd: Status 404 returned error can't find the container with id 07f236f0d771b1916b27fb5d0810385ea3efdc051768b75ed99a6dbe98b7a3cd Oct 30 20:49:30.030611 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:30.030573 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" event={"ID":"209affb4-e050-418c-b522-d120e03a8f0c","Type":"ContainerStarted","Data":"07f236f0d771b1916b27fb5d0810385ea3efdc051768b75ed99a6dbe98b7a3cd"} Oct 30 20:49:31.039006 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:31.038974 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" event={"ID":"209affb4-e050-418c-b522-d120e03a8f0c","Type":"ContainerStarted","Data":"b9708db984b68a2be418160c55c3c1c67a12f023726fc2229d159de217f1baf1"} Oct 30 20:49:31.039427 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:31.039060 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:31.158984 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:31.158933 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" podStartSLOduration=2.370307806 podStartE2EDuration="3.158921182s" totalImagesPullingTime="788.613376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:31.156576398 +0000 UTC m=+1334.083133317" watchObservedRunningTime="2026-10-30 20:49:31.158921182 +0000 UTC m=+1334.085478097" Oct 30 20:49:33.620888 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:33.620853 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc"] Oct 30 20:49:33.624957 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:33.624940 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:33.684743 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:33.684704 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc"] Oct 30 20:49:33.704789 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:33.704767 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-78ca07a3-serving-cert\"" Oct 30 20:49:33.768784 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:33.768751 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-maas-api-tls\") pod \"maas-api-e2e-ait-78ca07a3-75846fd895-2dflc\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:33.769040 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:33.769019 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzr68\" (UniqueName: \"kubernetes.io/projected/05e493f6-5c07-4c26-912c-2dae89676150-kube-api-access-hzr68\") pod \"maas-api-e2e-ait-78ca07a3-75846fd895-2dflc\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:33.769115 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:33.769080 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e493f6-5c07-4c26-912c-2dae89676150-openshift-service-ca\") pod \"maas-api-e2e-ait-78ca07a3-75846fd895-2dflc\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:33.769203 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:33.769187 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-metrics-certs\") pod \"maas-api-e2e-ait-78ca07a3-75846fd895-2dflc\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:33.870648 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:33.870610 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzr68\" (UniqueName: \"kubernetes.io/projected/05e493f6-5c07-4c26-912c-2dae89676150-kube-api-access-hzr68\") pod \"maas-api-e2e-ait-78ca07a3-75846fd895-2dflc\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:33.870843 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:33.870677 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e493f6-5c07-4c26-912c-2dae89676150-openshift-service-ca\") pod \"maas-api-e2e-ait-78ca07a3-75846fd895-2dflc\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:33.870843 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:33.870768 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-metrics-certs\") pod \"maas-api-e2e-ait-78ca07a3-75846fd895-2dflc\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:33.870843 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:33.870838 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-maas-api-tls\") pod \"maas-api-e2e-ait-78ca07a3-75846fd895-2dflc\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:33.871026 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:49:33.870970 5755 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-serving-cert: secret "maas-api-e2e-ait-78ca07a3-serving-cert" not found Oct 30 20:49:33.871070 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:49:33.871044 5755 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-maas-api-tls podName:05e493f6-5c07-4c26-912c-2dae89676150 nodeName:}" failed. No retries permitted until 2026-10-30 20:49:34.371021938 +0000 UTC m=+1337.297578845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-maas-api-tls") pod "maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" (UID: "05e493f6-5c07-4c26-912c-2dae89676150") : secret "maas-api-e2e-ait-78ca07a3-serving-cert" not found Oct 30 20:49:33.871367 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:33.871350 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e493f6-5c07-4c26-912c-2dae89676150-openshift-service-ca\") pod \"maas-api-e2e-ait-78ca07a3-75846fd895-2dflc\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:33.873497 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:33.873478 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-metrics-certs\") pod \"maas-api-e2e-ait-78ca07a3-75846fd895-2dflc\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:33.930263 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:33.930231 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzr68\" (UniqueName: \"kubernetes.io/projected/05e493f6-5c07-4c26-912c-2dae89676150-kube-api-access-hzr68\") pod \"maas-api-e2e-ait-78ca07a3-75846fd895-2dflc\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:34.377540 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:34.377506 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-maas-api-tls\") pod \"maas-api-e2e-ait-78ca07a3-75846fd895-2dflc\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:34.380203 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:34.380179 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-maas-api-tls\") pod \"maas-api-e2e-ait-78ca07a3-75846fd895-2dflc\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:34.537513 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:34.537485 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:34.936860 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:49:34.936810 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e493f6_5c07_4c26_912c_2dae89676150.slice/crio-3eeda8dec64fd61b1605529d724ce7f199367b189ad49d779f9e7535f32321d4 WatchSource:0}: Error finding container 3eeda8dec64fd61b1605529d724ce7f199367b189ad49d779f9e7535f32321d4: Status 404 returned error can't find the container with id 3eeda8dec64fd61b1605529d724ce7f199367b189ad49d779f9e7535f32321d4 Oct 30 20:49:34.937963 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:34.937930 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc"] Oct 30 20:49:35.067265 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:35.067226 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" event={"ID":"05e493f6-5c07-4c26-912c-2dae89676150","Type":"ContainerStarted","Data":"3eeda8dec64fd61b1605529d724ce7f199367b189ad49d779f9e7535f32321d4"} Oct 30 20:49:36.076453 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:36.076315 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" event={"ID":"05e493f6-5c07-4c26-912c-2dae89676150","Type":"ContainerStarted","Data":"58e8c162f9b4a9700f90abf89a715fc3ce9978e3cfbfe7f1b96120fefe41b3d4"} Oct 30 20:49:36.076453 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:36.076446 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:36.197135 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:36.197084 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" podStartSLOduration=2.389789047 podStartE2EDuration="3.197067193s" totalImagesPullingTime="807.278146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:36.192852086 +0000 UTC m=+1339.119409029" watchObservedRunningTime="2026-10-30 20:49:36.197067193 +0000 UTC m=+1339.123624110" Oct 30 20:49:37.052328 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:37.052302 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:49:38.848849 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:38.848817 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks"] Oct 30 20:49:38.853024 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:38.853006 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:49:38.941226 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:38.941151 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-maas-api-tls\") pod \"maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:49:38.941226 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:38.941187 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756c5191-71c8-4178-92c2-e10a6ecf0b83-openshift-service-ca\") pod \"maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:49:38.941418 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:38.941399 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-metrics-certs\") pod \"maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:49:38.941498 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:38.941481 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzsdv\" (UniqueName: \"kubernetes.io/projected/756c5191-71c8-4178-92c2-e10a6ecf0b83-kube-api-access-lzsdv\") pod \"maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:49:38.957220 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:38.957191 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-bd92310f-serving-cert\"" Oct 30 20:49:38.965919 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:38.965887 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks"] Oct 30 20:49:39.042833 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:39.042802 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-maas-api-tls\") pod \"maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:49:39.042991 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:39.042835 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756c5191-71c8-4178-92c2-e10a6ecf0b83-openshift-service-ca\") pod \"maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:49:39.042991 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:49:39.042964 5755 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-serving-cert: secret "maas-api-e2e-derive-bd92310f-serving-cert" not found Oct 30 20:49:39.043063 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:39.043006 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-metrics-certs\") pod \"maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:49:39.043063 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:49:39.043024 5755 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-maas-api-tls podName:756c5191-71c8-4178-92c2-e10a6ecf0b83 nodeName:}" failed. No retries permitted until 2026-10-30 20:49:39.54300829 +0000 UTC m=+1342.469565200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-maas-api-tls") pod "maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" (UID: "756c5191-71c8-4178-92c2-e10a6ecf0b83") : secret "maas-api-e2e-derive-bd92310f-serving-cert" not found Oct 30 20:49:39.043137 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:39.043110 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzsdv\" (UniqueName: \"kubernetes.io/projected/756c5191-71c8-4178-92c2-e10a6ecf0b83-kube-api-access-lzsdv\") pod \"maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:49:39.043664 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:39.043642 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756c5191-71c8-4178-92c2-e10a6ecf0b83-openshift-service-ca\") pod \"maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:49:39.045417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:39.045401 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-metrics-certs\") pod \"maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:49:39.120542 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:39.120505 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzsdv\" (UniqueName: \"kubernetes.io/projected/756c5191-71c8-4178-92c2-e10a6ecf0b83-kube-api-access-lzsdv\") pod \"maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:49:39.552719 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:39.552688 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-maas-api-tls\") pod \"maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:49:39.555312 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:39.555275 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-maas-api-tls\") pod \"maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:49:39.766614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:39.766582 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:49:40.144902 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:40.144874 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks"] Oct 30 20:49:40.145474 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:49:40.145376 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756c5191_71c8_4178_92c2_e10a6ecf0b83.slice/crio-d47cf13b9b8b1d8f31b61ef95c3b9133ca3277f9af659aed9fce7ab742e9e1dd WatchSource:0}: Error finding container d47cf13b9b8b1d8f31b61ef95c3b9133ca3277f9af659aed9fce7ab742e9e1dd: Status 404 returned error can't find the container with id d47cf13b9b8b1d8f31b61ef95c3b9133ca3277f9af659aed9fce7ab742e9e1dd Oct 30 20:49:41.128925 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:41.128880 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" event={"ID":"756c5191-71c8-4178-92c2-e10a6ecf0b83","Type":"ContainerStarted","Data":"0674f52ecb80ec9c28bbcd519f0b202908b02e6e8f829bb57879410192a96009"} Oct 30 20:49:41.129126 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:41.128940 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" event={"ID":"756c5191-71c8-4178-92c2-e10a6ecf0b83","Type":"ContainerStarted","Data":"d47cf13b9b8b1d8f31b61ef95c3b9133ca3277f9af659aed9fce7ab742e9e1dd"} Oct 30 20:49:41.129544 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:41.129519 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:49:41.248954 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:41.248895 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" podStartSLOduration=2.48496141 podStartE2EDuration="3.248875377s" totalImagesPullingTime="763.913967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:41.24602883 +0000 UTC m=+1344.172585758" watchObservedRunningTime="2026-10-30 20:49:41.248875377 +0000 UTC m=+1344.175432293" Oct 30 20:49:41.258919 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:49:41.258879 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:49:42.092582 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:42.092551 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:49:47.148572 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:49:47.148545 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:50:11.254753 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:50:11.254667 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:50:27.594486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.594441 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6"] Oct 30 20:50:27.600113 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.600089 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.652905 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.652882 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-78ca07a3-openshift-default-dockercfg-2f4mk\"" Oct 30 20:50:27.705260 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.705234 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istiod-ca-cert\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.705441 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.705352 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-data\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.705441 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.705373 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-podinfo\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.705594 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.705469 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8p2l\" (UniqueName: \"kubernetes.io/projected/e0b47b0d-bafd-44b6-b585-d24e1641e10d-kube-api-access-c8p2l\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.705594 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.705580 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-workload-certs\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.705666 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.705654 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-credential-socket\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.705714 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.705702 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-workload-socket\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.705758 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.705734 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-token\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.705801 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.705771 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-envoy\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.708927 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.708902 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6"] Oct 30 20:50:27.807199 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.807163 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-workload-socket\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.807199 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.807207 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-token\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.807370 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.807243 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-envoy\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.807370 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.807329 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istiod-ca-cert\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.807509 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.807415 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-data\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.807509 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.807441 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-podinfo\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.807509 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.807472 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8p2l\" (UniqueName: \"kubernetes.io/projected/e0b47b0d-bafd-44b6-b585-d24e1641e10d-kube-api-access-c8p2l\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.807657 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.807527 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-workload-certs\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.807657 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.807608 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-credential-socket\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.807657 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.807641 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-workload-socket\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.807811 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.807742 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-data\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.807953 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.807934 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-workload-certs\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.807999 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.807962 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-credential-socket\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.808105 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.808084 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istiod-ca-cert\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.809646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.809625 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-envoy\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.810036 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.810018 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-podinfo\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.865988 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.865925 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-token\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.866136 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.866115 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8p2l\" (UniqueName: \"kubernetes.io/projected/e0b47b0d-bafd-44b6-b585-d24e1641e10d-kube-api-access-c8p2l\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:27.914151 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:27.914128 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:28.346846 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:28.346822 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6"] Oct 30 20:50:28.349124 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:50:28.349101 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b47b0d_bafd_44b6_b585_d24e1641e10d.slice/crio-08f3ad5d837530f0892b304bb0a8f94e766d8b5e60e698a350c2a67f1db857ee WatchSource:0}: Error finding container 08f3ad5d837530f0892b304bb0a8f94e766d8b5e60e698a350c2a67f1db857ee: Status 404 returned error can't find the container with id 08f3ad5d837530f0892b304bb0a8f94e766d8b5e60e698a350c2a67f1db857ee Oct 30 20:50:28.352338 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:28.352305 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:50:28.352441 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:28.352368 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:50:28.352441 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:28.352430 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:50:28.489878 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:28.489839 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" event={"ID":"e0b47b0d-bafd-44b6-b585-d24e1641e10d","Type":"ContainerStarted","Data":"4808736c5324b181cc1efe276c3df68e388b8e2f4c9e68735c6c240cf3d37ef3"} Oct 30 20:50:28.490061 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:28.489904 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" event={"ID":"e0b47b0d-bafd-44b6-b585-d24e1641e10d","Type":"ContainerStarted","Data":"08f3ad5d837530f0892b304bb0a8f94e766d8b5e60e698a350c2a67f1db857ee"} Oct 30 20:50:28.615113 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:28.615012 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" podStartSLOduration=1.6149987289999999 podStartE2EDuration="1.614998729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:28.611785386 +0000 UTC m=+1391.538342319" watchObservedRunningTime="2026-10-30 20:50:28.614998729 +0000 UTC m=+1391.541555648" Oct 30 20:50:28.914636 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:28.914540 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:29.916705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:29.916669 5755 patch_prober.go:29] interesting pod/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:50:29.917150 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:29.916732 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" podUID="e0b47b0d-bafd-44b6-b585-d24e1641e10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:50:30.915348 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:30.915318 5755 patch_prober.go:29] interesting pod/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:50:30.915527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:30.915367 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" podUID="e0b47b0d-bafd-44b6-b585-d24e1641e10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:50:31.914953 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:31.914916 5755 patch_prober.go:29] interesting pod/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:50:31.915332 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:31.914970 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" podUID="e0b47b0d-bafd-44b6-b585-d24e1641e10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:50:32.915151 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:32.915115 5755 patch_prober.go:29] interesting pod/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:50:32.915574 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:32.915191 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" podUID="e0b47b0d-bafd-44b6-b585-d24e1641e10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:50:33.914861 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:33.914830 5755 patch_prober.go:29] interesting pod/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:50:33.915034 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:33.914889 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" podUID="e0b47b0d-bafd-44b6-b585-d24e1641e10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:50:34.919453 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:34.919419 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:34.919969 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:34.919950 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:34.920745 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:50:34.920724 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:50:41.255825 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:50:41.255787 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:51:06.117619 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.117584 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7"] Oct 30 20:51:06.121756 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.121739 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.176575 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.176542 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hsn45\"" Oct 30 20:51:06.225697 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.225662 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.225905 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.225715 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.225905 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.225863 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpm45\" (UniqueName: \"kubernetes.io/projected/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-kube-api-access-lpm45\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.225988 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.225955 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.226040 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.226026 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.226083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.226061 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.226122 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.226112 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.226165 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.226130 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.226165 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.226161 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.324602 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.324570 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7"] Oct 30 20:51:06.327627 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.327597 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.327789 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.327647 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.327941 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.327910 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpm45\" (UniqueName: \"kubernetes.io/projected/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-kube-api-access-lpm45\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.328037 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.328022 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.328126 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.328104 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.328204 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.328145 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.328269 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.328211 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.328269 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.328243 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.328367 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.328268 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.328828 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.328479 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.328828 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.328519 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.328828 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.328793 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.329027 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.328942 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.329319 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.329298 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.331879 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.331854 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.332903 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.332864 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.387278 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.387189 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.387461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.387310 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpm45\" (UniqueName: \"kubernetes.io/projected/9ee1db28-bf77-435f-b92f-d6dbd06a5e6a-kube-api-access-lpm45\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l95n7\" (UID: \"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.436203 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.436165 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:06.889638 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:51:06.889607 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee1db28_bf77_435f_b92f_d6dbd06a5e6a.slice/crio-1ce9a3a095605e6d07e2e70f41fc35a6d52bf8a935fa041baa79cb8ea14dba77 WatchSource:0}: Error finding container 1ce9a3a095605e6d07e2e70f41fc35a6d52bf8a935fa041baa79cb8ea14dba77: Status 404 returned error can't find the container with id 1ce9a3a095605e6d07e2e70f41fc35a6d52bf8a935fa041baa79cb8ea14dba77 Oct 30 20:51:06.889890 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.889866 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7"] Oct 30 20:51:06.892323 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.892290 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:51:06.892436 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.892353 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:51:06.892436 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:06.892402 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:51:07.801500 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:07.801452 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" event={"ID":"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a","Type":"ContainerStarted","Data":"397361abcf1e25a178a0336a917102713b107f2868d351192cb86c6bd221105a"} Oct 30 20:51:07.801935 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:07.801560 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" event={"ID":"9ee1db28-bf77-435f-b92f-d6dbd06a5e6a","Type":"ContainerStarted","Data":"1ce9a3a095605e6d07e2e70f41fc35a6d52bf8a935fa041baa79cb8ea14dba77"} Oct 30 20:51:07.997426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:07.997349 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" podStartSLOduration=1.997336168 podStartE2EDuration="1.997336168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:07.994953408 +0000 UTC m=+1430.921510350" watchObservedRunningTime="2026-10-30 20:51:07.997336168 +0000 UTC m=+1430.923893083" Oct 30 20:51:08.436700 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:08.436668 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:08.438469 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:08.438446 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-l95n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:51:08.438576 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:08.438494 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" podUID="9ee1db28-bf77-435f-b92f-d6dbd06a5e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:51:09.436846 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:09.436809 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-l95n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:51:09.437354 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:09.436871 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" podUID="9ee1db28-bf77-435f-b92f-d6dbd06a5e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:51:10.436562 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:10.436525 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-l95n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:51:10.436751 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:10.436584 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" podUID="9ee1db28-bf77-435f-b92f-d6dbd06a5e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:51:11.256817 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:51:11.256705 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:51:11.436674 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:11.436638 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-l95n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:51:11.436851 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:11.436690 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" podUID="9ee1db28-bf77-435f-b92f-d6dbd06a5e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:51:12.436698 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:12.436662 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-l95n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:51:12.437181 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:12.436722 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" podUID="9ee1db28-bf77-435f-b92f-d6dbd06a5e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:51:13.436942 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:13.436905 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-l95n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:51:13.437366 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:13.436971 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" podUID="9ee1db28-bf77-435f-b92f-d6dbd06a5e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:51:14.436581 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:14.436548 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-l95n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:51:14.436743 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:14.436606 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" podUID="9ee1db28-bf77-435f-b92f-d6dbd06a5e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:51:15.481728 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:15.481696 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:15.482329 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:15.482288 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:15.483510 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:15.483483 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" Oct 30 20:51:25.868751 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:25.868668 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s"] Oct 30 20:51:25.869289 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:25.868899 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" podUID="d0a0ee40-e313-40a2-9f98-865817e6f0cd" containerName="maas-api" containerID="cri-o://85107efaf22f6266a7a2fbb30f4345c48e9ad7ac6b3cd7c80674f9f992398d29" gracePeriod=30 Oct 30 20:51:26.200961 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.200928 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:51:26.368996 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.368953 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-metrics-certs\") pod \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " Oct 30 20:51:26.369197 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.369158 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a0ee40-e313-40a2-9f98-865817e6f0cd-kube-api-access-dqbwq\" (UniqueName: \"kubernetes.io/projected/d0a0ee40-e313-40a2-9f98-865817e6f0cd-kube-api-access-dqbwq\") pod \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " Oct 30 20:51:26.369269 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.369257 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-maas-api-tls\") pod \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " Oct 30 20:51:26.369328 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.369308 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a0ee40-e313-40a2-9f98-865817e6f0cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a0ee40-e313-40a2-9f98-865817e6f0cd-openshift-service-ca\") pod \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\" (UID: \"d0a0ee40-e313-40a2-9f98-865817e6f0cd\") " Oct 30 20:51:26.375038 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.372982 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-maas-api-tls" pod "d0a0ee40-e313-40a2-9f98-865817e6f0cd" (UID: "d0a0ee40-e313-40a2-9f98-865817e6f0cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:26.375038 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.373709 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a0ee40-e313-40a2-9f98-865817e6f0cd-openshift-service-ca" pod "d0a0ee40-e313-40a2-9f98-865817e6f0cd" (UID: "d0a0ee40-e313-40a2-9f98-865817e6f0cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:26.376646 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.376614 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-metrics-certs" pod "d0a0ee40-e313-40a2-9f98-865817e6f0cd" (UID: "d0a0ee40-e313-40a2-9f98-865817e6f0cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:26.380099 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.380050 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a0ee40-e313-40a2-9f98-865817e6f0cd-kube-api-access-dqbwq" pod "d0a0ee40-e313-40a2-9f98-865817e6f0cd" (UID: "d0a0ee40-e313-40a2-9f98-865817e6f0cd"). InnerVolumeSpecName "kube-api-access-dqbwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:26.471889 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.471862 5755 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-metrics-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:26.471889 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.471888 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqbwq\" (UniqueName: \"kubernetes.io/projected/d0a0ee40-e313-40a2-9f98-865817e6f0cd-kube-api-access-dqbwq\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:26.472130 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.471901 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a0ee40-e313-40a2-9f98-865817e6f0cd-maas-api-tls\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:26.472130 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.471909 5755 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a0ee40-e313-40a2-9f98-865817e6f0cd-openshift-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:26.940310 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.940223 5755 generic.go:356] "Generic (PLEG): container finished" podID="d0a0ee40-e313-40a2-9f98-865817e6f0cd" containerID="85107efaf22f6266a7a2fbb30f4345c48e9ad7ac6b3cd7c80674f9f992398d29" exitCode=0 Oct 30 20:51:26.940310 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.940271 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" Oct 30 20:51:26.940848 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.940310 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" event={"ID":"d0a0ee40-e313-40a2-9f98-865817e6f0cd","Type":"ContainerDied","Data":"85107efaf22f6266a7a2fbb30f4345c48e9ad7ac6b3cd7c80674f9f992398d29"} Oct 30 20:51:26.940848 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.940369 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s" event={"ID":"d0a0ee40-e313-40a2-9f98-865817e6f0cd","Type":"ContainerDied","Data":"9fcf37b4ed66bfcb3d0cbe3ea4e39e66753e1e1a7d6be642cec1fd0ad2447498"} Oct 30 20:51:26.940848 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.940415 5755 scope.go:122] "RemoveContainer" containerID="85107efaf22f6266a7a2fbb30f4345c48e9ad7ac6b3cd7c80674f9f992398d29" Oct 30 20:51:26.951762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.951745 5755 scope.go:122] "RemoveContainer" containerID="85107efaf22f6266a7a2fbb30f4345c48e9ad7ac6b3cd7c80674f9f992398d29" Oct 30 20:51:26.952075 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:51:26.952057 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85107efaf22f6266a7a2fbb30f4345c48e9ad7ac6b3cd7c80674f9f992398d29\": container with ID starting with 85107efaf22f6266a7a2fbb30f4345c48e9ad7ac6b3cd7c80674f9f992398d29 not found: ID does not exist" containerID="85107efaf22f6266a7a2fbb30f4345c48e9ad7ac6b3cd7c80674f9f992398d29" Oct 30 20:51:26.952121 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:26.952086 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85107efaf22f6266a7a2fbb30f4345c48e9ad7ac6b3cd7c80674f9f992398d29"} err="failed to get container status \"85107efaf22f6266a7a2fbb30f4345c48e9ad7ac6b3cd7c80674f9f992398d29\": rpc error: code = NotFound desc = could not find container \"85107efaf22f6266a7a2fbb30f4345c48e9ad7ac6b3cd7c80674f9f992398d29\": container with ID starting with 85107efaf22f6266a7a2fbb30f4345c48e9ad7ac6b3cd7c80674f9f992398d29 not found: ID does not exist" Oct 30 20:51:27.112977 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:27.112941 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s"] Oct 30 20:51:27.161292 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:27.161261 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dc0e9f64-7cfbccf565-pkc5s"] Oct 30 20:51:27.213815 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:27.213742 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a0ee40-e313-40a2-9f98-865817e6f0cd" path="/var/lib/kubelet/pods/d0a0ee40-e313-40a2-9f98-865817e6f0cd/volumes" Oct 30 20:51:32.799212 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:32.799174 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn"] Oct 30 20:51:32.799718 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:32.799442 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" podUID="209affb4-e050-418c-b522-d120e03a8f0c" containerName="maas-api" containerID="cri-o://b9708db984b68a2be418160c55c3c1c67a12f023726fc2229d159de217f1baf1" gracePeriod=30 Oct 30 20:51:33.009566 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:33.009125 5755 generic.go:356] "Generic (PLEG): container finished" podID="209affb4-e050-418c-b522-d120e03a8f0c" containerID="b9708db984b68a2be418160c55c3c1c67a12f023726fc2229d159de217f1baf1" exitCode=0 Oct 30 20:51:33.009566 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:33.009427 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" event={"ID":"209affb4-e050-418c-b522-d120e03a8f0c","Type":"ContainerDied","Data":"b9708db984b68a2be418160c55c3c1c67a12f023726fc2229d159de217f1baf1"} Oct 30 20:51:33.117257 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:33.117222 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:51:33.166728 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:33.166690 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-maas-api-tls\") pod \"209affb4-e050-418c-b522-d120e03a8f0c\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " Oct 30 20:51:33.166939 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:33.166751 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209affb4-e050-418c-b522-d120e03a8f0c-kube-api-access-dbx9x\" (UniqueName: \"kubernetes.io/projected/209affb4-e050-418c-b522-d120e03a8f0c-kube-api-access-dbx9x\") pod \"209affb4-e050-418c-b522-d120e03a8f0c\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " Oct 30 20:51:33.166939 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:33.166821 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-metrics-certs\") pod \"209affb4-e050-418c-b522-d120e03a8f0c\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " Oct 30 20:51:33.166939 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:33.166876 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/209affb4-e050-418c-b522-d120e03a8f0c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209affb4-e050-418c-b522-d120e03a8f0c-openshift-service-ca\") pod \"209affb4-e050-418c-b522-d120e03a8f0c\" (UID: \"209affb4-e050-418c-b522-d120e03a8f0c\") " Oct 30 20:51:33.167672 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:33.167232 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209affb4-e050-418c-b522-d120e03a8f0c-openshift-service-ca" pod "209affb4-e050-418c-b522-d120e03a8f0c" (UID: "209affb4-e050-418c-b522-d120e03a8f0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:33.167672 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:33.167634 5755 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209affb4-e050-418c-b522-d120e03a8f0c-openshift-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:33.170024 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:33.169593 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-maas-api-tls" pod "209affb4-e050-418c-b522-d120e03a8f0c" (UID: "209affb4-e050-418c-b522-d120e03a8f0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:33.170024 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:33.169634 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209affb4-e050-418c-b522-d120e03a8f0c-kube-api-access-dbx9x" pod "209affb4-e050-418c-b522-d120e03a8f0c" (UID: "209affb4-e050-418c-b522-d120e03a8f0c"). InnerVolumeSpecName "kube-api-access-dbx9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:33.170537 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:33.170512 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-metrics-certs" pod "209affb4-e050-418c-b522-d120e03a8f0c" (UID: "209affb4-e050-418c-b522-d120e03a8f0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:33.269604 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:33.269519 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-maas-api-tls\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:33.269604 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:33.269565 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbx9x\" (UniqueName: \"kubernetes.io/projected/209affb4-e050-418c-b522-d120e03a8f0c-kube-api-access-dbx9x\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:33.269604 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:33.269583 5755 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/209affb4-e050-418c-b522-d120e03a8f0c-metrics-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:34.018584 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.018513 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" Oct 30 20:51:34.018584 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.018532 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn" event={"ID":"209affb4-e050-418c-b522-d120e03a8f0c","Type":"ContainerDied","Data":"07f236f0d771b1916b27fb5d0810385ea3efdc051768b75ed99a6dbe98b7a3cd"} Oct 30 20:51:34.021972 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.018612 5755 scope.go:122] "RemoveContainer" containerID="b9708db984b68a2be418160c55c3c1c67a12f023726fc2229d159de217f1baf1" Oct 30 20:51:34.190991 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.190960 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn"] Oct 30 20:51:34.240673 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.240636 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9782c31-f96989bd8-tskqn"] Oct 30 20:51:34.263765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.263729 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc"] Oct 30 20:51:34.263990 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.263964 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" podUID="05e493f6-5c07-4c26-912c-2dae89676150" containerName="maas-api" containerID="cri-o://58e8c162f9b4a9700f90abf89a715fc3ce9978e3cfbfe7f1b96120fefe41b3d4" gracePeriod=30 Oct 30 20:51:34.619044 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.619013 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:51:34.727947 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.727908 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-maas-api-tls\") pod \"05e493f6-5c07-4c26-912c-2dae89676150\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " Oct 30 20:51:34.728147 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.728052 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e493f6-5c07-4c26-912c-2dae89676150-kube-api-access-hzr68\" (UniqueName: \"kubernetes.io/projected/05e493f6-5c07-4c26-912c-2dae89676150-kube-api-access-hzr68\") pod \"05e493f6-5c07-4c26-912c-2dae89676150\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " Oct 30 20:51:34.728147 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.728102 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05e493f6-5c07-4c26-912c-2dae89676150-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e493f6-5c07-4c26-912c-2dae89676150-openshift-service-ca\") pod \"05e493f6-5c07-4c26-912c-2dae89676150\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " Oct 30 20:51:34.728242 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.728151 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-metrics-certs\") pod \"05e493f6-5c07-4c26-912c-2dae89676150\" (UID: \"05e493f6-5c07-4c26-912c-2dae89676150\") " Oct 30 20:51:34.729742 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.729634 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e493f6-5c07-4c26-912c-2dae89676150-openshift-service-ca" pod "05e493f6-5c07-4c26-912c-2dae89676150" (UID: "05e493f6-5c07-4c26-912c-2dae89676150"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:34.731743 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.731716 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-metrics-certs" pod "05e493f6-5c07-4c26-912c-2dae89676150" (UID: "05e493f6-5c07-4c26-912c-2dae89676150"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:34.732246 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.732222 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-maas-api-tls" pod "05e493f6-5c07-4c26-912c-2dae89676150" (UID: "05e493f6-5c07-4c26-912c-2dae89676150"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:34.732664 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.732644 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e493f6-5c07-4c26-912c-2dae89676150-kube-api-access-hzr68" pod "05e493f6-5c07-4c26-912c-2dae89676150" (UID: "05e493f6-5c07-4c26-912c-2dae89676150"). InnerVolumeSpecName "kube-api-access-hzr68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:34.831337 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.831242 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzr68\" (UniqueName: \"kubernetes.io/projected/05e493f6-5c07-4c26-912c-2dae89676150-kube-api-access-hzr68\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:34.831337 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.831280 5755 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e493f6-5c07-4c26-912c-2dae89676150-openshift-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:34.831337 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.831296 5755 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-metrics-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:34.831337 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:34.831311 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05e493f6-5c07-4c26-912c-2dae89676150-maas-api-tls\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:35.027990 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:35.027955 5755 generic.go:356] "Generic (PLEG): container finished" podID="05e493f6-5c07-4c26-912c-2dae89676150" containerID="58e8c162f9b4a9700f90abf89a715fc3ce9978e3cfbfe7f1b96120fefe41b3d4" exitCode=0 Oct 30 20:51:35.028469 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:35.028010 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" Oct 30 20:51:35.028469 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:35.028040 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" event={"ID":"05e493f6-5c07-4c26-912c-2dae89676150","Type":"ContainerDied","Data":"58e8c162f9b4a9700f90abf89a715fc3ce9978e3cfbfe7f1b96120fefe41b3d4"} Oct 30 20:51:35.028469 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:35.028141 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc" event={"ID":"05e493f6-5c07-4c26-912c-2dae89676150","Type":"ContainerDied","Data":"3eeda8dec64fd61b1605529d724ce7f199367b189ad49d779f9e7535f32321d4"} Oct 30 20:51:35.028469 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:35.028173 5755 scope.go:122] "RemoveContainer" containerID="58e8c162f9b4a9700f90abf89a715fc3ce9978e3cfbfe7f1b96120fefe41b3d4" Oct 30 20:51:35.042485 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:35.042463 5755 scope.go:122] "RemoveContainer" containerID="58e8c162f9b4a9700f90abf89a715fc3ce9978e3cfbfe7f1b96120fefe41b3d4" Oct 30 20:51:35.042890 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:51:35.042864 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e8c162f9b4a9700f90abf89a715fc3ce9978e3cfbfe7f1b96120fefe41b3d4\": container with ID starting with 58e8c162f9b4a9700f90abf89a715fc3ce9978e3cfbfe7f1b96120fefe41b3d4 not found: ID does not exist" containerID="58e8c162f9b4a9700f90abf89a715fc3ce9978e3cfbfe7f1b96120fefe41b3d4" Oct 30 20:51:35.042965 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:35.042901 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e8c162f9b4a9700f90abf89a715fc3ce9978e3cfbfe7f1b96120fefe41b3d4"} err="failed to get container status \"58e8c162f9b4a9700f90abf89a715fc3ce9978e3cfbfe7f1b96120fefe41b3d4\": rpc error: code = NotFound desc = could not find container \"58e8c162f9b4a9700f90abf89a715fc3ce9978e3cfbfe7f1b96120fefe41b3d4\": container with ID starting with 58e8c162f9b4a9700f90abf89a715fc3ce9978e3cfbfe7f1b96120fefe41b3d4 not found: ID does not exist" Oct 30 20:51:35.197882 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:35.197788 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc"] Oct 30 20:51:35.213654 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:35.213623 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="209affb4-e050-418c-b522-d120e03a8f0c" path="/var/lib/kubelet/pods/209affb4-e050-418c-b522-d120e03a8f0c/volumes" Oct 30 20:51:35.247092 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:35.247056 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78ca07a3-75846fd895-2dflc"] Oct 30 20:51:36.124981 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:36.124945 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks"] Oct 30 20:51:36.125540 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:36.125181 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" podUID="756c5191-71c8-4178-92c2-e10a6ecf0b83" containerName="maas-api" containerID="cri-o://0674f52ecb80ec9c28bbcd519f0b202908b02e6e8f829bb57879410192a96009" gracePeriod=30 Oct 30 20:51:36.434861 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:36.434422 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:51:36.566922 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:36.566877 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-metrics-certs\") pod \"756c5191-71c8-4178-92c2-e10a6ecf0b83\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " Oct 30 20:51:36.567112 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:36.566970 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756c5191-71c8-4178-92c2-e10a6ecf0b83-kube-api-access-lzsdv\" (UniqueName: \"kubernetes.io/projected/756c5191-71c8-4178-92c2-e10a6ecf0b83-kube-api-access-lzsdv\") pod \"756c5191-71c8-4178-92c2-e10a6ecf0b83\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " Oct 30 20:51:36.567112 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:36.567003 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-maas-api-tls\") pod \"756c5191-71c8-4178-92c2-e10a6ecf0b83\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " Oct 30 20:51:36.567237 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:36.567133 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/756c5191-71c8-4178-92c2-e10a6ecf0b83-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756c5191-71c8-4178-92c2-e10a6ecf0b83-openshift-service-ca\") pod \"756c5191-71c8-4178-92c2-e10a6ecf0b83\" (UID: \"756c5191-71c8-4178-92c2-e10a6ecf0b83\") " Oct 30 20:51:36.571406 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:36.568368 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756c5191-71c8-4178-92c2-e10a6ecf0b83-openshift-service-ca" pod "756c5191-71c8-4178-92c2-e10a6ecf0b83" (UID: "756c5191-71c8-4178-92c2-e10a6ecf0b83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:36.624688 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:36.624413 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-maas-api-tls" pod "756c5191-71c8-4178-92c2-e10a6ecf0b83" (UID: "756c5191-71c8-4178-92c2-e10a6ecf0b83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:36.624688 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:36.624483 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756c5191-71c8-4178-92c2-e10a6ecf0b83-kube-api-access-lzsdv" pod "756c5191-71c8-4178-92c2-e10a6ecf0b83" (UID: "756c5191-71c8-4178-92c2-e10a6ecf0b83"). InnerVolumeSpecName "kube-api-access-lzsdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:36.624688 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:36.624565 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-metrics-certs" pod "756c5191-71c8-4178-92c2-e10a6ecf0b83" (UID: "756c5191-71c8-4178-92c2-e10a6ecf0b83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:36.669852 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:36.669758 5755 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-metrics-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:36.669852 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:36.669791 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzsdv\" (UniqueName: \"kubernetes.io/projected/756c5191-71c8-4178-92c2-e10a6ecf0b83-kube-api-access-lzsdv\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:36.669852 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:36.669808 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756c5191-71c8-4178-92c2-e10a6ecf0b83-maas-api-tls\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:36.669852 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:36.669826 5755 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756c5191-71c8-4178-92c2-e10a6ecf0b83-openshift-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:37.046509 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:37.046472 5755 generic.go:356] "Generic (PLEG): container finished" podID="756c5191-71c8-4178-92c2-e10a6ecf0b83" containerID="0674f52ecb80ec9c28bbcd519f0b202908b02e6e8f829bb57879410192a96009" exitCode=0 Oct 30 20:51:37.046725 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:37.046527 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" Oct 30 20:51:37.046725 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:37.046562 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" event={"ID":"756c5191-71c8-4178-92c2-e10a6ecf0b83","Type":"ContainerDied","Data":"0674f52ecb80ec9c28bbcd519f0b202908b02e6e8f829bb57879410192a96009"} Oct 30 20:51:37.046725 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:37.046626 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks" event={"ID":"756c5191-71c8-4178-92c2-e10a6ecf0b83","Type":"ContainerDied","Data":"d47cf13b9b8b1d8f31b61ef95c3b9133ca3277f9af659aed9fce7ab742e9e1dd"} Oct 30 20:51:37.046725 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:37.046653 5755 scope.go:122] "RemoveContainer" containerID="0674f52ecb80ec9c28bbcd519f0b202908b02e6e8f829bb57879410192a96009" Oct 30 20:51:37.060875 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:37.060855 5755 scope.go:122] "RemoveContainer" containerID="0674f52ecb80ec9c28bbcd519f0b202908b02e6e8f829bb57879410192a96009" Oct 30 20:51:37.061189 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:51:37.061170 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0674f52ecb80ec9c28bbcd519f0b202908b02e6e8f829bb57879410192a96009\": container with ID starting with 0674f52ecb80ec9c28bbcd519f0b202908b02e6e8f829bb57879410192a96009 not found: ID does not exist" containerID="0674f52ecb80ec9c28bbcd519f0b202908b02e6e8f829bb57879410192a96009" Oct 30 20:51:37.061252 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:37.061203 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0674f52ecb80ec9c28bbcd519f0b202908b02e6e8f829bb57879410192a96009"} err="failed to get container status \"0674f52ecb80ec9c28bbcd519f0b202908b02e6e8f829bb57879410192a96009\": rpc error: code = NotFound desc = could not find container \"0674f52ecb80ec9c28bbcd519f0b202908b02e6e8f829bb57879410192a96009\": container with ID starting with 0674f52ecb80ec9c28bbcd519f0b202908b02e6e8f829bb57879410192a96009 not found: ID does not exist" Oct 30 20:51:37.245536 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:37.243545 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05e493f6-5c07-4c26-912c-2dae89676150" path="/var/lib/kubelet/pods/05e493f6-5c07-4c26-912c-2dae89676150/volumes" Oct 30 20:51:37.245536 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:37.244085 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks"] Oct 30 20:51:37.268556 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:37.268506 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd92310f-5d9b6b64d7-jwxks"] Oct 30 20:51:38.836215 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:38.836181 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6"] Oct 30 20:51:38.836677 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:38.836419 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" podUID="e0b47b0d-bafd-44b6-b585-d24e1641e10d" containerName="istio-proxy" containerID="cri-o://4808736c5324b181cc1efe276c3df68e388b8e2f4c9e68735c6c240cf3d37ef3" gracePeriod=30 Oct 30 20:51:39.214533 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:39.214501 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="756c5191-71c8-4178-92c2-e10a6ecf0b83" path="/var/lib/kubelet/pods/756c5191-71c8-4178-92c2-e10a6ecf0b83/volumes" Oct 30 20:51:41.257866 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:51:41.257828 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:51:44.130853 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.130819 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:51:44.143121 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.143091 5755 generic.go:356] "Generic (PLEG): container finished" podID="e0b47b0d-bafd-44b6-b585-d24e1641e10d" containerID="4808736c5324b181cc1efe276c3df68e388b8e2f4c9e68735c6c240cf3d37ef3" exitCode=0 Oct 30 20:51:44.143267 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.143132 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" Oct 30 20:51:44.143267 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.143182 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" event={"ID":"e0b47b0d-bafd-44b6-b585-d24e1641e10d","Type":"ContainerDied","Data":"4808736c5324b181cc1efe276c3df68e388b8e2f4c9e68735c6c240cf3d37ef3"} Oct 30 20:51:44.143267 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.143233 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6" event={"ID":"e0b47b0d-bafd-44b6-b585-d24e1641e10d","Type":"ContainerDied","Data":"08f3ad5d837530f0892b304bb0a8f94e766d8b5e60e698a350c2a67f1db857ee"} Oct 30 20:51:44.143267 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.143255 5755 scope.go:122] "RemoveContainer" containerID="4808736c5324b181cc1efe276c3df68e388b8e2f4c9e68735c6c240cf3d37ef3" Oct 30 20:51:44.154640 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.154617 5755 scope.go:122] "RemoveContainer" containerID="4808736c5324b181cc1efe276c3df68e388b8e2f4c9e68735c6c240cf3d37ef3" Oct 30 20:51:44.154958 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:51:44.154937 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4808736c5324b181cc1efe276c3df68e388b8e2f4c9e68735c6c240cf3d37ef3\": container with ID starting with 4808736c5324b181cc1efe276c3df68e388b8e2f4c9e68735c6c240cf3d37ef3 not found: ID does not exist" containerID="4808736c5324b181cc1efe276c3df68e388b8e2f4c9e68735c6c240cf3d37ef3" Oct 30 20:51:44.155039 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.154968 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4808736c5324b181cc1efe276c3df68e388b8e2f4c9e68735c6c240cf3d37ef3"} err="failed to get container status \"4808736c5324b181cc1efe276c3df68e388b8e2f4c9e68735c6c240cf3d37ef3\": rpc error: code = NotFound desc = could not find container \"4808736c5324b181cc1efe276c3df68e388b8e2f4c9e68735c6c240cf3d37ef3\": container with ID starting with 4808736c5324b181cc1efe276c3df68e388b8e2f4c9e68735c6c240cf3d37ef3 not found: ID does not exist" Oct 30 20:51:44.279357 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.279326 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-data\") pod \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " Oct 30 20:51:44.279600 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.279368 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-envoy\") pod \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " Oct 30 20:51:44.279600 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.279446 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-workload-socket\") pod \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " Oct 30 20:51:44.279600 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.279534 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istiod-ca-cert\") pod \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " Oct 30 20:51:44.279600 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.279570 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-token\" (UniqueName: \"kubernetes.io/projected/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-token\") pod \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " Oct 30 20:51:44.279600 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.279596 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-credential-socket\") pod \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " Oct 30 20:51:44.279855 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.279701 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-data" pod "e0b47b0d-bafd-44b6-b585-d24e1641e10d" (UID: "e0b47b0d-bafd-44b6-b585-d24e1641e10d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:51:44.279855 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.279746 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-podinfo\") pod \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " Oct 30 20:51:44.279855 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.279834 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-workload-certs\") pod \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " Oct 30 20:51:44.280008 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.279925 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b47b0d-bafd-44b6-b585-d24e1641e10d-kube-api-access-c8p2l\" (UniqueName: \"kubernetes.io/projected/e0b47b0d-bafd-44b6-b585-d24e1641e10d-kube-api-access-c8p2l\") pod \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\" (UID: \"e0b47b0d-bafd-44b6-b585-d24e1641e10d\") " Oct 30 20:51:44.280008 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.279931 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-workload-socket" pod "e0b47b0d-bafd-44b6-b585-d24e1641e10d" (UID: "e0b47b0d-bafd-44b6-b585-d24e1641e10d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:51:44.280157 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.280027 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istiod-ca-cert" pod "e0b47b0d-bafd-44b6-b585-d24e1641e10d" (UID: "e0b47b0d-bafd-44b6-b585-d24e1641e10d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:44.280157 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.280121 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-workload-certs" pod "e0b47b0d-bafd-44b6-b585-d24e1641e10d" (UID: "e0b47b0d-bafd-44b6-b585-d24e1641e10d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:51:44.281193 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.280911 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:44.281193 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.281047 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:44.281193 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.281069 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:44.281193 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.281085 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:44.281520 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.281498 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-credential-socket" pod "e0b47b0d-bafd-44b6-b585-d24e1641e10d" (UID: "e0b47b0d-bafd-44b6-b585-d24e1641e10d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:51:44.283025 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.282996 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b47b0d-bafd-44b6-b585-d24e1641e10d-kube-api-access-c8p2l" pod "e0b47b0d-bafd-44b6-b585-d24e1641e10d" (UID: "e0b47b0d-bafd-44b6-b585-d24e1641e10d"). InnerVolumeSpecName "kube-api-access-c8p2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:44.283127 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.283046 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-token" pod "e0b47b0d-bafd-44b6-b585-d24e1641e10d" (UID: "e0b47b0d-bafd-44b6-b585-d24e1641e10d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:44.283127 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.283072 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-podinfo" pod "e0b47b0d-bafd-44b6-b585-d24e1641e10d" (UID: "e0b47b0d-bafd-44b6-b585-d24e1641e10d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:51:44.283522 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.283502 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-envoy" pod "e0b47b0d-bafd-44b6-b585-d24e1641e10d" (UID: "e0b47b0d-bafd-44b6-b585-d24e1641e10d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:51:44.383484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.382569 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:44.383484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.382603 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:44.383484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.382618 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b47b0d-bafd-44b6-b585-d24e1641e10d-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:44.383484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.382631 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b47b0d-bafd-44b6-b585-d24e1641e10d-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:44.383484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.382645 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8p2l\" (UniqueName: \"kubernetes.io/projected/e0b47b0d-bafd-44b6-b585-d24e1641e10d-kube-api-access-c8p2l\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:44.393609 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.393574 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9"] Oct 30 20:51:44.394493 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.394465 5755 cpu_manager.go:414] "containerMap: removing container" podUID="d0a0ee40-e313-40a2-9f98-865817e6f0cd" containerName="maas-api" Oct 30 20:51:44.394493 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.394490 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a0ee40-e313-40a2-9f98-865817e6f0cd" containerName="maas-api" Oct 30 20:51:44.394723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.394518 5755 cpu_manager.go:414] "containerMap: removing container" podUID="05e493f6-5c07-4c26-912c-2dae89676150" containerName="maas-api" Oct 30 20:51:44.394723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.394527 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e493f6-5c07-4c26-912c-2dae89676150" containerName="maas-api" Oct 30 20:51:44.394723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.394538 5755 cpu_manager.go:414] "containerMap: removing container" podUID="756c5191-71c8-4178-92c2-e10a6ecf0b83" containerName="maas-api" Oct 30 20:51:44.394723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.394547 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756c5191-71c8-4178-92c2-e10a6ecf0b83" containerName="maas-api" Oct 30 20:51:44.394723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.394583 5755 cpu_manager.go:414] "containerMap: removing container" podUID="e0b47b0d-bafd-44b6-b585-d24e1641e10d" containerName="istio-proxy" Oct 30 20:51:44.394723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.394590 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b47b0d-bafd-44b6-b585-d24e1641e10d" containerName="istio-proxy" Oct 30 20:51:44.394723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.394628 5755 cpu_manager.go:414] "containerMap: removing container" podUID="209affb4-e050-418c-b522-d120e03a8f0c" containerName="maas-api" Oct 30 20:51:44.394723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.394635 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209affb4-e050-418c-b522-d120e03a8f0c" containerName="maas-api" Oct 30 20:51:44.395108 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.394822 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b47b0d-bafd-44b6-b585-d24e1641e10d" containerName="istio-proxy" Oct 30 20:51:44.395108 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.394839 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="209affb4-e050-418c-b522-d120e03a8f0c" containerName="maas-api" Oct 30 20:51:44.395108 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.394854 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a0ee40-e313-40a2-9f98-865817e6f0cd" containerName="maas-api" Oct 30 20:51:44.395108 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.394868 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e493f6-5c07-4c26-912c-2dae89676150" containerName="maas-api" Oct 30 20:51:44.395108 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.394878 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="756c5191-71c8-4178-92c2-e10a6ecf0b83" containerName="maas-api" Oct 30 20:51:44.400097 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.400076 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.456180 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.456157 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 20:51:44.456305 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.456248 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 20:51:44.456934 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.456916 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 20:51:44.458306 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.458290 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jmh9r\"" Oct 30 20:51:44.484602 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.484572 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4872bc-be2d-4689-87aa-702ab0659e8c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.484748 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.484618 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.484748 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.484652 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.484946 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.484923 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.485019 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.485003 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.485093 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.485079 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mh8b\" (UniqueName: \"kubernetes.io/projected/4d4872bc-be2d-4689-87aa-702ab0659e8c-kube-api-access-7mh8b\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.485147 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.485123 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.487611 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.487589 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9"] Oct 30 20:51:44.586937 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.586901 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4872bc-be2d-4689-87aa-702ab0659e8c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.586937 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.586944 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.587169 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.586962 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.587169 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.587125 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.587271 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.587174 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.587271 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.587222 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mh8b\" (UniqueName: \"kubernetes.io/projected/4d4872bc-be2d-4689-87aa-702ab0659e8c-kube-api-access-7mh8b\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.587271 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.587259 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.588036 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.587965 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.589397 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.589363 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4872bc-be2d-4689-87aa-702ab0659e8c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.589706 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.589686 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.590020 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.590002 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.590083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.590036 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.644410 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.644295 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6"] Oct 30 20:51:44.647166 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.647141 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.647311 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.647292 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mh8b\" (UniqueName: \"kubernetes.io/projected/4d4872bc-be2d-4689-87aa-702ab0659e8c-kube-api-access-7mh8b\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vmrs9\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:44.648176 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.648153 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-lcpg6"] Oct 30 20:51:44.714248 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:44.714212 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:45.154101 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:45.154053 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9"] Oct 30 20:51:45.158018 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:51:45.157988 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4872bc_be2d_4689_87aa_702ab0659e8c.slice/crio-90df6af32e290e84d01c806e3daed0152c35859cd83863513fca4e7b1fe0e98b WatchSource:0}: Error finding container 90df6af32e290e84d01c806e3daed0152c35859cd83863513fca4e7b1fe0e98b: Status 404 returned error can't find the container with id 90df6af32e290e84d01c806e3daed0152c35859cd83863513fca4e7b1fe0e98b Oct 30 20:51:45.213967 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:45.213933 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b47b0d-bafd-44b6-b585-d24e1641e10d" path="/var/lib/kubelet/pods/e0b47b0d-bafd-44b6-b585-d24e1641e10d/volumes" Oct 30 20:51:46.171560 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:46.171522 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" event={"ID":"4d4872bc-be2d-4689-87aa-702ab0659e8c","Type":"ContainerStarted","Data":"90df6af32e290e84d01c806e3daed0152c35859cd83863513fca4e7b1fe0e98b"} Oct 30 20:51:49.099520 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:49.099483 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:51:49.099930 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:49.099548 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:51:49.200319 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:49.200271 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" event={"ID":"4d4872bc-be2d-4689-87aa-702ab0659e8c","Type":"ContainerStarted","Data":"151e133f3ca7308bfd192957ce8f0dfe55678f0e62236741fec812673895112f"} Oct 30 20:51:49.200527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:49.200428 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:51:49.357490 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:49.357359 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" podStartSLOduration=1.419725965 podStartE2EDuration="5.357344338s" totalImagesPullingTime="3.937618373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:49.354473108 +0000 UTC m=+1472.281030051" watchObservedRunningTime="2026-10-30 20:51:49.357344338 +0000 UTC m=+1472.283901253" Oct 30 20:51:50.211624 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:50.211591 5755 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vmrs9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:51:50.212110 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:50.211644 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" podUID="4d4872bc-be2d-4689-87aa-702ab0659e8c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:51:52.065160 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:52.065120 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf"] Oct 30 20:51:52.109120 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:52.109075 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8484f78-k2xzf"] Oct 30 20:51:53.213762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:53.213728 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="207b3254-36dd-450e-841a-bc771bf4ac3a" path="/var/lib/kubelet/pods/207b3254-36dd-450e-841a-bc771bf4ac3a/volumes" Oct 30 20:51:53.214118 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:51:53.214062 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 20:52:11.256481 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:52:11.256442 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:52:19.084863 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:19.084831 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj"] Oct 30 20:52:19.133966 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:19.133926 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3795a660-6dgjj"] Oct 30 20:52:19.212784 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:19.212753 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd51c45-4e46-4841-acdb-073d7e3441fc" path="/var/lib/kubelet/pods/dfd51c45-4e46-4841-acdb-073d7e3441fc/volumes" Oct 30 20:52:38.644410 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:38.644364 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc"] Oct 30 20:52:38.648571 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:38.648554 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc" Oct 30 20:52:38.706009 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:38.705974 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc"] Oct 30 20:52:38.751096 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:38.751066 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:52:38.751245 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:38.751121 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:52:38.783436 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:38.783377 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c05a66-tqnxc\" (UID: \"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc" Oct 30 20:52:38.783436 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:38.783441 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7zjl\" (UniqueName: \"kubernetes.io/projected/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-kube-api-access-m7zjl\") pod \"maas-api-revoke-sub-80c05a66-tqnxc\" (UID: \"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc" Oct 30 20:52:38.783627 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:38.783462 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c05a66-tqnxc\" (UID: \"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc" Oct 30 20:52:38.885505 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:38.885475 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c05a66-tqnxc\" (UID: \"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc" Oct 30 20:52:38.885505 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:38.885512 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7zjl\" (UniqueName: \"kubernetes.io/projected/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-kube-api-access-m7zjl\") pod \"maas-api-revoke-sub-80c05a66-tqnxc\" (UID: \"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc" Oct 30 20:52:38.885756 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:38.885534 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c05a66-tqnxc\" (UID: \"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc" Oct 30 20:52:38.886118 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:38.886097 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c05a66-tqnxc\" (UID: \"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc" Oct 30 20:52:38.888068 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:38.888044 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c05a66-tqnxc\" (UID: \"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc" Oct 30 20:52:38.943600 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:38.943534 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7zjl\" (UniqueName: \"kubernetes.io/projected/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-kube-api-access-m7zjl\") pod \"maas-api-revoke-sub-80c05a66-tqnxc\" (UID: \"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc" Oct 30 20:52:38.962077 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:38.962051 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc" Oct 30 20:52:39.348599 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:52:39.348565 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24db4576_f3e1_475a_aa4b_ea0b7d66fbf7.slice/crio-ee3edef1fc4a99de9b062040650b6b17745bedcaa9606b3cb0c86ad42daed405 WatchSource:0}: Error finding container ee3edef1fc4a99de9b062040650b6b17745bedcaa9606b3cb0c86ad42daed405: Status 404 returned error can't find the container with id ee3edef1fc4a99de9b062040650b6b17745bedcaa9606b3cb0c86ad42daed405 Oct 30 20:52:39.348857 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:39.348835 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc"] Oct 30 20:52:39.576605 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:39.576572 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc" event={"ID":"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7","Type":"ContainerStarted","Data":"ee3edef1fc4a99de9b062040650b6b17745bedcaa9606b3cb0c86ad42daed405"} Oct 30 20:52:40.586637 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:40.586557 5755 generic.go:356] "Generic (PLEG): container finished" podID="24db4576-f3e1-475a-aa4b-ea0b7d66fbf7" containerID="c9083b269030f952b333116908b90bd6388c92937747f15a504e005113aabbcf" exitCode=0 Oct 30 20:52:40.586985 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:40.586637 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc" event={"ID":"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7","Type":"ContainerDied","Data":"c9083b269030f952b333116908b90bd6388c92937747f15a504e005113aabbcf"} Oct 30 20:52:41.255647 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:52:41.255604 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:52:41.753168 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:41.753146 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc" Oct 30 20:52:41.831482 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:41.831456 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-kube-api-access-m7zjl\" (UniqueName: \"kubernetes.io/projected/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-kube-api-access-m7zjl\") pod \"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7\" (UID: \"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7\") " Oct 30 20:52:41.831618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:41.831492 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-maas-api-xks-ca\") pod \"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7\" (UID: \"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7\") " Oct 30 20:52:41.831618 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:41.831603 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-maas-api-openshift-ca\") pod \"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7\" (UID: \"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7\") " Oct 30 20:52:41.831904 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:41.831875 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-maas-api-openshift-ca" pod "24db4576-f3e1-475a-aa4b-ea0b7d66fbf7" (UID: "24db4576-f3e1-475a-aa4b-ea0b7d66fbf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:41.832235 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:41.832177 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:52:41.833675 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:41.833653 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-maas-api-xks-ca" pod "24db4576-f3e1-475a-aa4b-ea0b7d66fbf7" (UID: "24db4576-f3e1-475a-aa4b-ea0b7d66fbf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:41.833811 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:41.833792 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-kube-api-access-m7zjl" pod "24db4576-f3e1-475a-aa4b-ea0b7d66fbf7" (UID: "24db4576-f3e1-475a-aa4b-ea0b7d66fbf7"). InnerVolumeSpecName "kube-api-access-m7zjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:41.933244 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:41.933167 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7zjl\" (UniqueName: \"kubernetes.io/projected/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-kube-api-access-m7zjl\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:52:41.933244 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:41.933192 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:52:42.055981 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:42.055944 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4"] Oct 30 20:52:42.060672 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:42.060645 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f82c1a-nx8x4"] Oct 30 20:52:42.602123 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:42.602096 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc" Oct 30 20:52:42.602367 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:42.602125 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc" event={"ID":"24db4576-f3e1-475a-aa4b-ea0b7d66fbf7","Type":"ContainerDied","Data":"ee3edef1fc4a99de9b062040650b6b17745bedcaa9606b3cb0c86ad42daed405"} Oct 30 20:52:42.602367 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:42.602176 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee3edef1fc4a99de9b062040650b6b17745bedcaa9606b3cb0c86ad42daed405" Oct 30 20:52:43.213198 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:43.213164 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15171cf4-7349-4232-a057-2d28352faae1" path="/var/lib/kubelet/pods/15171cf4-7349-4232-a057-2d28352faae1/volumes" Oct 30 20:52:49.120702 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:49.120670 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52"] Oct 30 20:52:49.170037 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:49.170000 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f"] Oct 30 20:52:49.173234 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:49.173212 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb95c8ca-5mm52"] Oct 30 20:52:49.176057 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:49.176038 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eee1174-2wk8f"] Oct 30 20:52:49.213970 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:49.213942 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25fc72ef-aea6-44a7-9077-40ecd86c99a2" path="/var/lib/kubelet/pods/25fc72ef-aea6-44a7-9077-40ecd86c99a2/volumes" Oct 30 20:52:49.214442 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:49.214429 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf" path="/var/lib/kubelet/pods/4b894288-6c1d-4247-b4ff-1a7d3e2d4dcf/volumes" Oct 30 20:52:51.946238 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:51.946199 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252"] Oct 30 20:52:51.946999 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:51.946978 5755 cpu_manager.go:414] "containerMap: removing container" podUID="24db4576-f3e1-475a-aa4b-ea0b7d66fbf7" containerName="revoke-keys" Oct 30 20:52:51.947096 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:51.947001 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24db4576-f3e1-475a-aa4b-ea0b7d66fbf7" containerName="revoke-keys" Oct 30 20:52:51.947198 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:51.947184 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="24db4576-f3e1-475a-aa4b-ea0b7d66fbf7" containerName="revoke-keys" Oct 30 20:52:51.952052 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:51.952031 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:51.979469 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:51.979425 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:51.979625 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:51.979472 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:51.979703 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:51.979651 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2965df53-329d-4971-893e-3c60b72762c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:51.979703 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:51.979686 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:51.979830 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:51.979706 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2965df53-329d-4971-893e-3c60b72762c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:51.979830 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:51.979723 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2965df53-329d-4971-893e-3c60b72762c5-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:51.979830 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:51.979824 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvp2j\" (UniqueName: \"kubernetes.io/projected/2965df53-329d-4971-893e-3c60b72762c5-kube-api-access-jvp2j\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:51.980006 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:51.979876 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:51.980064 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:51.980003 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:51.998800 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:51.998761 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7"] Oct 30 20:52:52.004276 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.004238 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.057120 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.057091 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252"] Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.081904 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2965df53-329d-4971-893e-3c60b72762c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.081984 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.082014 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.082044 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2965df53-329d-4971-893e-3c60b72762c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.082070 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2965df53-329d-4971-893e-3c60b72762c5-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.082101 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.082165 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64jjc\" (UniqueName: \"kubernetes.io/projected/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-kube-api-access-64jjc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.082205 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvp2j\" (UniqueName: \"kubernetes.io/projected/2965df53-329d-4971-893e-3c60b72762c5-kube-api-access-jvp2j\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.082262 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.082307 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.082360 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.082418 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.082446 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.082552 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.082581 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.082606 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.082643 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.082690 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.083075 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.083232 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2965df53-329d-4971-893e-3c60b72762c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.083606 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.084068 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.089426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.084565 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.093096 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.093069 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2965df53-329d-4971-893e-3c60b72762c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.093892 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.093868 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.124290 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.124256 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7"] Oct 30 20:52:52.143606 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.143573 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvp2j\" (UniqueName: \"kubernetes.io/projected/2965df53-329d-4971-893e-3c60b72762c5-kube-api-access-jvp2j\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.147191 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.147033 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2965df53-329d-4971-893e-3c60b72762c5-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vk252\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.183878 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.183849 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.184029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.183925 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.184029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.183960 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.184029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.183985 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.184214 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.184070 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.184214 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.184096 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.184320 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.184300 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.184400 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.184352 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.184462 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.184407 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64jjc\" (UniqueName: \"kubernetes.io/projected/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-kube-api-access-64jjc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.185029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.185008 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.185565 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.185546 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.186133 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.186061 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.186223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.186207 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.186481 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.186455 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.189379 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.188285 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.190106 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.190083 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.243817 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.243787 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.243817 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.243813 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64jjc\" (UniqueName: \"kubernetes.io/projected/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-kube-api-access-64jjc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lm9p7\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.264950 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.264922 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:52.325987 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.324820 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:52.702057 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.702029 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252"] Oct 30 20:52:52.703463 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:52:52.703434 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2965df53_329d_4971_893e_3c60b72762c5.slice/crio-2863087bc524a73f73b8fb02d71f934df594a398370c28df3ad94d0965070822 WatchSource:0}: Error finding container 2863087bc524a73f73b8fb02d71f934df594a398370c28df3ad94d0965070822: Status 404 returned error can't find the container with id 2863087bc524a73f73b8fb02d71f934df594a398370c28df3ad94d0965070822 Oct 30 20:52:52.706553 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.706513 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:52.706657 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.706598 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:52.706657 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.706642 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:52.758540 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:52:52.758514 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e15faa4_f00e_4316_9b78_fe0b40a09ac2.slice/crio-ad3673d277f4ebc0524c1fd0a2be3c49313496d85cab1ac97b8f8e928c64c4b4 WatchSource:0}: Error finding container ad3673d277f4ebc0524c1fd0a2be3c49313496d85cab1ac97b8f8e928c64c4b4: Status 404 returned error can't find the container with id ad3673d277f4ebc0524c1fd0a2be3c49313496d85cab1ac97b8f8e928c64c4b4 Oct 30 20:52:52.758709 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.758557 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7"] Oct 30 20:52:52.761207 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.761168 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:52.761306 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.761255 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:52.761306 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:52.761298 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:53.717838 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:53.717760 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" event={"ID":"2965df53-329d-4971-893e-3c60b72762c5","Type":"ContainerStarted","Data":"7c06707a7b2b6d9a3683dcecab74d71866c7ce33e48e986ca22021dfd3218d33"} Oct 30 20:52:53.720914 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:53.720152 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" event={"ID":"2965df53-329d-4971-893e-3c60b72762c5","Type":"ContainerStarted","Data":"2863087bc524a73f73b8fb02d71f934df594a398370c28df3ad94d0965070822"} Oct 30 20:52:53.723359 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:53.723332 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" event={"ID":"4e15faa4-f00e-4316-9b78-fe0b40a09ac2","Type":"ContainerStarted","Data":"f9533248ebd674201f131e65b0660472914b651df7c2d52856233e47f6257793"} Oct 30 20:52:53.723825 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:53.723799 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" event={"ID":"4e15faa4-f00e-4316-9b78-fe0b40a09ac2","Type":"ContainerStarted","Data":"ad3673d277f4ebc0524c1fd0a2be3c49313496d85cab1ac97b8f8e928c64c4b4"} Oct 30 20:52:53.945187 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:53.945135 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" podStartSLOduration=2.945120469 podStartE2EDuration="2.945120469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:53.895011387 +0000 UTC m=+1536.821568322" watchObservedRunningTime="2026-10-30 20:52:53.945120469 +0000 UTC m=+1536.871677651" Oct 30 20:52:54.025324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:54.025165 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" podStartSLOduration=3.025145881 podStartE2EDuration="3.025145881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:54.020770618 +0000 UTC m=+1536.947327559" watchObservedRunningTime="2026-10-30 20:52:54.025145881 +0000 UTC m=+1536.951702796" Oct 30 20:52:54.265405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:54.265362 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:52:54.266444 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:54.266422 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vk252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:52:54.266535 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:54.266464 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" podUID="2965df53-329d-4971-893e-3c60b72762c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 20:52:54.342875 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:54.342774 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:52:54.345622 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:54.344492 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lm9p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:52:54.345622 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:54.344553 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" podUID="4e15faa4-f00e-4316-9b78-fe0b40a09ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:52:55.266233 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:55.266205 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vk252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:52:55.266653 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:55.266250 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" podUID="2965df53-329d-4971-893e-3c60b72762c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 20:52:55.326057 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:55.326023 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lm9p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:52:55.326235 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:55.326071 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" podUID="4e15faa4-f00e-4316-9b78-fe0b40a09ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:52:56.265527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:56.265494 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vk252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:52:56.265677 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:56.265546 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" podUID="2965df53-329d-4971-893e-3c60b72762c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 20:52:56.325875 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:56.325836 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lm9p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:52:56.326255 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:56.325890 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" podUID="4e15faa4-f00e-4316-9b78-fe0b40a09ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:52:57.065316 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:57.065282 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq"] Oct 30 20:52:57.123102 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:57.123046 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d14ed15-9f4qq"] Oct 30 20:52:57.227320 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:57.227285 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc" path="/var/lib/kubelet/pods/f16bcfa8-bec1-484d-8cc8-a40ac3e6e4cc/volumes" Oct 30 20:52:57.265489 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:57.265460 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vk252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:52:57.265621 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:57.265519 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" podUID="2965df53-329d-4971-893e-3c60b72762c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 20:52:57.326022 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:57.325929 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lm9p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:52:57.326022 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:57.325982 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" podUID="4e15faa4-f00e-4316-9b78-fe0b40a09ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:52:58.265898 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:58.265859 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vk252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:52:58.266067 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:58.265923 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" podUID="2965df53-329d-4971-893e-3c60b72762c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 20:52:58.325575 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:58.325537 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lm9p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:52:58.325747 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:58.325606 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" podUID="4e15faa4-f00e-4316-9b78-fe0b40a09ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:52:59.265979 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:59.265948 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vk252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:52:59.266369 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:59.265997 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" podUID="2965df53-329d-4971-893e-3c60b72762c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 20:52:59.326187 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:59.326154 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lm9p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:52:59.326344 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:52:59.326216 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" podUID="4e15faa4-f00e-4316-9b78-fe0b40a09ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:53:00.265296 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:00.265255 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vk252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:53:00.265486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:00.265332 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" podUID="2965df53-329d-4971-893e-3c60b72762c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 20:53:00.325723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:00.325688 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lm9p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:53:00.326076 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:00.325760 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" podUID="4e15faa4-f00e-4316-9b78-fe0b40a09ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:53:01.586855 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:01.586816 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:53:01.587649 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:01.587571 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:53:01.588686 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:01.588667 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 20:53:01.635025 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:01.634992 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:53:01.635656 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:01.635633 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:53:01.636941 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:01.636916 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 20:53:05.052421 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:05.052376 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf"] Oct 30 20:53:05.102358 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:05.102326 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e8ca55-kgspf"] Oct 30 20:53:05.212992 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:05.212962 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388b927b-777b-46d2-a43c-fc36dc22a7ef" path="/var/lib/kubelet/pods/388b927b-777b-46d2-a43c-fc36dc22a7ef/volumes" Oct 30 20:53:07.455040 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:07.454994 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" podUID="b1b5f93e01c03391b49a05418cff0a8e" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 20:53:08.052911 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:08.052873 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8"] Oct 30 20:53:08.102505 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:08.102472 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6200448-zwtx8"] Oct 30 20:53:09.212754 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:09.212719 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3e20f0-ee6c-4544-8bb4-25364ca6b6de" path="/var/lib/kubelet/pods/cb3e20f0-ee6c-4544-8bb4-25364ca6b6de/volumes" Oct 30 20:53:11.255965 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:53:11.255926 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:53:16.053281 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:16.053243 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6"] Oct 30 20:53:16.104205 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:16.104169 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3b1c816-lxvr6"] Oct 30 20:53:17.213783 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:17.213753 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db66f48e-1108-4674-b486-568a20c7e7de" path="/var/lib/kubelet/pods/db66f48e-1108-4674-b486-568a20c7e7de/volumes" Oct 30 20:53:17.764596 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:17.764564 5755 scope.go:122] "RemoveContainer" containerID="1efbfab64a5b6e351e54ed0794343942ae5b70255f30cd3efbc6c3fa08c9582d" Oct 30 20:53:17.775025 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:17.775005 5755 scope.go:122] "RemoveContainer" containerID="18627c0cdb1a71f7db6e385078123b709372171ee485256dc17d60952a683075" Oct 30 20:53:38.505898 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:38.505851 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 20:53:38.889490 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:38.889410 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9"] Oct 30 20:53:38.893670 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:38.893654 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" Oct 30 20:53:38.951122 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:38.951090 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9"] Oct 30 20:53:38.984020 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:38.983985 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1984d4d-e168-4633-96e2-7bca3856afc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9\" (UID: \"e1984d4d-e168-4633-96e2-7bca3856afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" Oct 30 20:53:38.984203 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:38.984033 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1984d4d-e168-4633-96e2-7bca3856afc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9\" (UID: \"e1984d4d-e168-4633-96e2-7bca3856afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" Oct 30 20:53:38.984300 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:38.984279 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9j65\" (UniqueName: \"kubernetes.io/projected/e1984d4d-e168-4633-96e2-7bca3856afc5-kube-api-access-k9j65\") pod \"maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9\" (UID: \"e1984d4d-e168-4633-96e2-7bca3856afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" Oct 30 20:53:39.001111 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:38.998692 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:53:39.001111 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:38.998713 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:53:39.085267 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:39.085236 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9j65\" (UniqueName: \"kubernetes.io/projected/e1984d4d-e168-4633-96e2-7bca3856afc5-kube-api-access-k9j65\") pod \"maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9\" (UID: \"e1984d4d-e168-4633-96e2-7bca3856afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" Oct 30 20:53:39.085455 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:39.085358 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1984d4d-e168-4633-96e2-7bca3856afc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9\" (UID: \"e1984d4d-e168-4633-96e2-7bca3856afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" Oct 30 20:53:39.085455 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:39.085395 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1984d4d-e168-4633-96e2-7bca3856afc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9\" (UID: \"e1984d4d-e168-4633-96e2-7bca3856afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" Oct 30 20:53:39.086105 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:39.086050 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1984d4d-e168-4633-96e2-7bca3856afc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9\" (UID: \"e1984d4d-e168-4633-96e2-7bca3856afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" Oct 30 20:53:39.088158 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:39.088139 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1984d4d-e168-4633-96e2-7bca3856afc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9\" (UID: \"e1984d4d-e168-4633-96e2-7bca3856afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" Oct 30 20:53:39.144091 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:39.144025 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9j65\" (UniqueName: \"kubernetes.io/projected/e1984d4d-e168-4633-96e2-7bca3856afc5-kube-api-access-k9j65\") pod \"maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9\" (UID: \"e1984d4d-e168-4633-96e2-7bca3856afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" Oct 30 20:53:39.208684 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:39.208650 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" Oct 30 20:53:39.614375 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:39.614347 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9"] Oct 30 20:53:39.615773 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:53:39.615747 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1984d4d_e168_4633_96e2_7bca3856afc5.slice/crio-f4a108028ef87963c6a0e7614821ad03f8a4f20d45b2f5a76359202e68dac678 WatchSource:0}: Error finding container f4a108028ef87963c6a0e7614821ad03f8a4f20d45b2f5a76359202e68dac678: Status 404 returned error can't find the container with id f4a108028ef87963c6a0e7614821ad03f8a4f20d45b2f5a76359202e68dac678 Oct 30 20:53:40.150219 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:40.150095 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" event={"ID":"e1984d4d-e168-4633-96e2-7bca3856afc5","Type":"ContainerStarted","Data":"54806a3f43fe07f38bb943fa7fa85a98d779faa2e126af08af77f0c00ffc5c65"} Oct 30 20:53:40.150219 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:40.150161 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" event={"ID":"e1984d4d-e168-4633-96e2-7bca3856afc5","Type":"ContainerStarted","Data":"f4a108028ef87963c6a0e7614821ad03f8a4f20d45b2f5a76359202e68dac678"} Oct 30 20:53:40.267991 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:40.267939 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" podStartSLOduration=2.267924968 podStartE2EDuration="2.267924968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:40.265779248 +0000 UTC m=+1583.192336163" watchObservedRunningTime="2026-10-30 20:53:40.267924968 +0000 UTC m=+1583.194481881" Oct 30 20:53:41.158804 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:41.158770 5755 generic.go:356] "Generic (PLEG): container finished" podID="e1984d4d-e168-4633-96e2-7bca3856afc5" containerID="54806a3f43fe07f38bb943fa7fa85a98d779faa2e126af08af77f0c00ffc5c65" exitCode=0 Oct 30 20:53:41.159173 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:41.158851 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" event={"ID":"e1984d4d-e168-4633-96e2-7bca3856afc5","Type":"ContainerDied","Data":"54806a3f43fe07f38bb943fa7fa85a98d779faa2e126af08af77f0c00ffc5c65"} Oct 30 20:53:41.259261 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:53:41.259226 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:53:42.310504 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:42.310479 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" Oct 30 20:53:42.437683 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:42.437609 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1984d4d-e168-4633-96e2-7bca3856afc5-kube-api-access-k9j65\" (UniqueName: \"kubernetes.io/projected/e1984d4d-e168-4633-96e2-7bca3856afc5-kube-api-access-k9j65\") pod \"e1984d4d-e168-4633-96e2-7bca3856afc5\" (UID: \"e1984d4d-e168-4633-96e2-7bca3856afc5\") " Oct 30 20:53:42.437877 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:42.437861 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1984d4d-e168-4633-96e2-7bca3856afc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1984d4d-e168-4633-96e2-7bca3856afc5-maas-api-openshift-ca\") pod \"e1984d4d-e168-4633-96e2-7bca3856afc5\" (UID: \"e1984d4d-e168-4633-96e2-7bca3856afc5\") " Oct 30 20:53:42.437990 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:42.437977 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1984d4d-e168-4633-96e2-7bca3856afc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1984d4d-e168-4633-96e2-7bca3856afc5-maas-api-xks-ca\") pod \"e1984d4d-e168-4633-96e2-7bca3856afc5\" (UID: \"e1984d4d-e168-4633-96e2-7bca3856afc5\") " Oct 30 20:53:42.438266 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:42.438236 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1984d4d-e168-4633-96e2-7bca3856afc5-maas-api-openshift-ca" pod "e1984d4d-e168-4633-96e2-7bca3856afc5" (UID: "e1984d4d-e168-4633-96e2-7bca3856afc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:42.438534 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:42.438517 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1984d4d-e168-4633-96e2-7bca3856afc5-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:42.440562 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:42.440535 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1984d4d-e168-4633-96e2-7bca3856afc5-kube-api-access-k9j65" pod "e1984d4d-e168-4633-96e2-7bca3856afc5" (UID: "e1984d4d-e168-4633-96e2-7bca3856afc5"). InnerVolumeSpecName "kube-api-access-k9j65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:53:42.441776 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:42.441751 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1984d4d-e168-4633-96e2-7bca3856afc5-maas-api-xks-ca" pod "e1984d4d-e168-4633-96e2-7bca3856afc5" (UID: "e1984d4d-e168-4633-96e2-7bca3856afc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:42.539711 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:42.539682 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1984d4d-e168-4633-96e2-7bca3856afc5-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:42.539711 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:42.539712 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9j65\" (UniqueName: \"kubernetes.io/projected/e1984d4d-e168-4633-96e2-7bca3856afc5-kube-api-access-k9j65\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:43.175710 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.175676 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" event={"ID":"e1984d4d-e168-4633-96e2-7bca3856afc5","Type":"ContainerDied","Data":"f4a108028ef87963c6a0e7614821ad03f8a4f20d45b2f5a76359202e68dac678"} Oct 30 20:53:43.175891 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.175731 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4a108028ef87963c6a0e7614821ad03f8a4f20d45b2f5a76359202e68dac678" Oct 30 20:53:43.175891 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.175687 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9" Oct 30 20:53:43.213550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.213525 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64"] Oct 30 20:53:43.745048 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.745015 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk"] Oct 30 20:53:43.745666 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.745652 5755 cpu_manager.go:414] "containerMap: removing container" podUID="e1984d4d-e168-4633-96e2-7bca3856afc5" containerName="revoke-keys" Oct 30 20:53:43.745731 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.745668 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1984d4d-e168-4633-96e2-7bca3856afc5" containerName="revoke-keys" Oct 30 20:53:43.745781 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.745773 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1984d4d-e168-4633-96e2-7bca3856afc5" containerName="revoke-keys" Oct 30 20:53:43.749256 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.749235 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" Oct 30 20:53:43.755843 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.755821 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db450192-3182-4d15-9af2-cb9875996024-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk\" (UID: \"db450192-3182-4d15-9af2-cb9875996024\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" Oct 30 20:53:43.755962 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.755932 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q5q9\" (UniqueName: \"kubernetes.io/projected/db450192-3182-4d15-9af2-cb9875996024-kube-api-access-4q5q9\") pod \"maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk\" (UID: \"db450192-3182-4d15-9af2-cb9875996024\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" Oct 30 20:53:43.756177 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.756153 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db450192-3182-4d15-9af2-cb9875996024-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk\" (UID: \"db450192-3182-4d15-9af2-cb9875996024\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" Oct 30 20:53:43.806283 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.806256 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:53:43.806589 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.806270 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:53:43.810844 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.810823 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk"] Oct 30 20:53:43.857547 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.857505 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db450192-3182-4d15-9af2-cb9875996024-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk\" (UID: \"db450192-3182-4d15-9af2-cb9875996024\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" Oct 30 20:53:43.857716 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.857696 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db450192-3182-4d15-9af2-cb9875996024-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk\" (UID: \"db450192-3182-4d15-9af2-cb9875996024\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" Oct 30 20:53:43.857771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.857741 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q5q9\" (UniqueName: \"kubernetes.io/projected/db450192-3182-4d15-9af2-cb9875996024-kube-api-access-4q5q9\") pod \"maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk\" (UID: \"db450192-3182-4d15-9af2-cb9875996024\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" Oct 30 20:53:43.858338 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.858312 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db450192-3182-4d15-9af2-cb9875996024-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk\" (UID: \"db450192-3182-4d15-9af2-cb9875996024\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" Oct 30 20:53:43.860265 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.860240 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db450192-3182-4d15-9af2-cb9875996024-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk\" (UID: \"db450192-3182-4d15-9af2-cb9875996024\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" Oct 30 20:53:43.920125 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:43.920089 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q5q9\" (UniqueName: \"kubernetes.io/projected/db450192-3182-4d15-9af2-cb9875996024-kube-api-access-4q5q9\") pod \"maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk\" (UID: \"db450192-3182-4d15-9af2-cb9875996024\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" Oct 30 20:53:44.063714 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:44.063635 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" Oct 30 20:53:44.448524 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:44.448500 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk"] Oct 30 20:53:44.449094 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:53:44.449070 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb450192_3182_4d15_9af2_cb9875996024.slice/crio-2225cc5e9725d647f37ce6f9ae28b95ef99e010f613d6f94b09f99a302d35872 WatchSource:0}: Error finding container 2225cc5e9725d647f37ce6f9ae28b95ef99e010f613d6f94b09f99a302d35872: Status 404 returned error can't find the container with id 2225cc5e9725d647f37ce6f9ae28b95ef99e010f613d6f94b09f99a302d35872 Oct 30 20:53:45.192526 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:45.192440 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" event={"ID":"db450192-3182-4d15-9af2-cb9875996024","Type":"ContainerStarted","Data":"8aa1aac9a46dafccdd497867baa3daaa15da11c31a6a0b6dbea99507b937d9a6"} Oct 30 20:53:45.192895 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:45.192535 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" event={"ID":"db450192-3182-4d15-9af2-cb9875996024","Type":"ContainerStarted","Data":"2225cc5e9725d647f37ce6f9ae28b95ef99e010f613d6f94b09f99a302d35872"} Oct 30 20:53:45.323737 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:45.323684 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" podStartSLOduration=2.323670348 podStartE2EDuration="2.323670348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:45.321592903 +0000 UTC m=+1588.248149855" watchObservedRunningTime="2026-10-30 20:53:45.323670348 +0000 UTC m=+1588.250227270" Oct 30 20:53:46.200486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.200444 5755 generic.go:356] "Generic (PLEG): container finished" podID="db450192-3182-4d15-9af2-cb9875996024" containerID="8aa1aac9a46dafccdd497867baa3daaa15da11c31a6a0b6dbea99507b937d9a6" exitCode=0 Oct 30 20:53:46.200895 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.200572 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" event={"ID":"db450192-3182-4d15-9af2-cb9875996024","Type":"ContainerDied","Data":"8aa1aac9a46dafccdd497867baa3daaa15da11c31a6a0b6dbea99507b937d9a6"} Oct 30 20:53:46.592038 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.591994 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v"] Oct 30 20:53:46.617292 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.617264 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.655788 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.655751 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v"] Oct 30 20:53:46.672602 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.672574 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-0702ff-openshift-default-dockercfg-krdfv\"" Oct 30 20:53:46.715517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.715481 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-workload-certs\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.715692 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.715616 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-workload-socket\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.715692 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.715678 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t2tv\" (UniqueName: \"kubernetes.io/projected/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-kube-api-access-9t2tv\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.715801 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.715775 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-data\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.715898 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.715883 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-envoy\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.715953 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.715931 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-podinfo\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.716000 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.715991 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-token\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.716034 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.716010 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istiod-ca-cert\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.716034 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.716027 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-credential-socket\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.817136 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.817098 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-workload-certs\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.817301 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.817213 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-workload-socket\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.817301 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.817291 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t2tv\" (UniqueName: \"kubernetes.io/projected/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-kube-api-access-9t2tv\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.817379 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.817338 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-data\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.817474 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.817459 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-envoy\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.817534 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.817492 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-podinfo\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.817590 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.817542 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-token\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.817590 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.817568 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istiod-ca-cert\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.817696 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.817595 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-credential-socket\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.817696 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.817611 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-workload-socket\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.817807 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.817790 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-data\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.817914 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.817616 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-workload-certs\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.818084 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.818057 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-credential-socket\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.818323 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.818298 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istiod-ca-cert\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.820230 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.820208 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-podinfo\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.820714 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.820697 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-envoy\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.878672 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.878604 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-token\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.878882 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.878716 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t2tv\" (UniqueName: \"kubernetes.io/projected/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-kube-api-access-9t2tv\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:46.979467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:46.954759 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:47.475449 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:47.475335 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" Oct 30 20:53:47.552153 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:47.551364 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:47.552153 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:47.551471 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:47.552153 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:47.551501 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:47.561317 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:47.556730 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db450192-3182-4d15-9af2-cb9875996024-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db450192-3182-4d15-9af2-cb9875996024-maas-api-xks-ca\") pod \"db450192-3182-4d15-9af2-cb9875996024\" (UID: \"db450192-3182-4d15-9af2-cb9875996024\") " Oct 30 20:53:47.561317 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:47.556822 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db450192-3182-4d15-9af2-cb9875996024-kube-api-access-4q5q9\" (UniqueName: \"kubernetes.io/projected/db450192-3182-4d15-9af2-cb9875996024-kube-api-access-4q5q9\") pod \"db450192-3182-4d15-9af2-cb9875996024\" (UID: \"db450192-3182-4d15-9af2-cb9875996024\") " Oct 30 20:53:47.561317 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:47.556909 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db450192-3182-4d15-9af2-cb9875996024-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db450192-3182-4d15-9af2-cb9875996024-maas-api-openshift-ca\") pod \"db450192-3182-4d15-9af2-cb9875996024\" (UID: \"db450192-3182-4d15-9af2-cb9875996024\") " Oct 30 20:53:47.561317 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:47.557221 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v"] Oct 30 20:53:47.561317 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:47.557507 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db450192-3182-4d15-9af2-cb9875996024-maas-api-openshift-ca" pod "db450192-3182-4d15-9af2-cb9875996024" (UID: "db450192-3182-4d15-9af2-cb9875996024"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:47.564523 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:47.561966 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db450192-3182-4d15-9af2-cb9875996024-kube-api-access-4q5q9" pod "db450192-3182-4d15-9af2-cb9875996024" (UID: "db450192-3182-4d15-9af2-cb9875996024"). InnerVolumeSpecName "kube-api-access-4q5q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:53:47.570275 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:47.566788 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db450192-3182-4d15-9af2-cb9875996024-maas-api-xks-ca" pod "db450192-3182-4d15-9af2-cb9875996024" (UID: "db450192-3182-4d15-9af2-cb9875996024"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:47.658653 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:47.658617 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db450192-3182-4d15-9af2-cb9875996024-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:47.658747 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:47.658661 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q5q9\" (UniqueName: \"kubernetes.io/projected/db450192-3182-4d15-9af2-cb9875996024-kube-api-access-4q5q9\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:47.658747 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:47.658679 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db450192-3182-4d15-9af2-cb9875996024-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:48.262875 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:48.262827 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" event={"ID":"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc","Type":"ContainerStarted","Data":"fdeb707e66f2412af9f6b08aa23b02a9377de6e5c81b415bfc8e94d0b105a02f"} Oct 30 20:53:48.263067 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:48.262897 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" event={"ID":"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc","Type":"ContainerStarted","Data":"d5aa16d1e690096144175964a81844d428b979a608692238adf8577ae6ea91d3"} Oct 30 20:53:48.264953 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:48.264922 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" Oct 30 20:53:48.265097 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:48.265010 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk" event={"ID":"db450192-3182-4d15-9af2-cb9875996024","Type":"ContainerDied","Data":"2225cc5e9725d647f37ce6f9ae28b95ef99e010f613d6f94b09f99a302d35872"} Oct 30 20:53:48.265097 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:48.265050 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2225cc5e9725d647f37ce6f9ae28b95ef99e010f613d6f94b09f99a302d35872" Oct 30 20:53:48.438858 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:48.438808 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" podStartSLOduration=2.438795839 podStartE2EDuration="2.438795839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:48.418081223 +0000 UTC m=+1591.344638161" watchObservedRunningTime="2026-10-30 20:53:48.438795839 +0000 UTC m=+1591.365352753" Oct 30 20:53:48.959835 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:48.959807 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:48.961611 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:48.961586 5755 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 20:53:48.961735 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:48.961630 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" podUID="ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 20:53:49.960587 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:49.960554 5755 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 20:53:49.960973 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:49.960612 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" podUID="ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 20:53:50.960372 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:50.960341 5755 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 20:53:50.960562 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:50.960414 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" podUID="ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 20:53:51.960744 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:51.960712 5755 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 20:53:51.961238 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:51.960768 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" podUID="ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 20:53:52.960785 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:52.960754 5755 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 20:53:52.961235 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:52.960838 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" podUID="ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 20:53:53.960169 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:53.960131 5755 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 20:53:53.960381 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:53.960201 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" podUID="ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 20:53:54.964544 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:54.964514 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:54.964914 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:54.964884 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:54.965620 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:54.965596 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:53:58.235718 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.235681 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" containerID="cri-o://48d4df7a0c35b7ede7363d91a660bbe248703e5ccc553ae7a3c39ea0361c9749" gracePeriod=45 Oct 30 20:53:58.372055 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.372017 5755 generic.go:356] "Generic (PLEG): container finished" podID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerID="48d4df7a0c35b7ede7363d91a660bbe248703e5ccc553ae7a3c39ea0361c9749" exitCode=0 Oct 30 20:53:58.372226 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.372081 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" event={"ID":"b2b57028-ba25-496d-834e-5433b95cc2a4","Type":"ContainerDied","Data":"48d4df7a0c35b7ede7363d91a660bbe248703e5ccc553ae7a3c39ea0361c9749"} Oct 30 20:53:58.504057 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.503973 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:53:58.649197 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.649163 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-model-cache\") pod \"b2b57028-ba25-496d-834e-5433b95cc2a4\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " Oct 30 20:53:58.649368 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.649223 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2b57028-ba25-496d-834e-5433b95cc2a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b57028-ba25-496d-834e-5433b95cc2a4-tls-certs\") pod \"b2b57028-ba25-496d-834e-5433b95cc2a4\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " Oct 30 20:53:58.649368 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.649302 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-home\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-home\") pod \"b2b57028-ba25-496d-834e-5433b95cc2a4\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " Oct 30 20:53:58.649368 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.649334 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-dshm\") pod \"b2b57028-ba25-496d-834e-5433b95cc2a4\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " Oct 30 20:53:58.649368 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.649355 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b57028-ba25-496d-834e-5433b95cc2a4-kube-api-access-gbzdx\" (UniqueName: \"kubernetes.io/projected/b2b57028-ba25-496d-834e-5433b95cc2a4-kube-api-access-gbzdx\") pod \"b2b57028-ba25-496d-834e-5433b95cc2a4\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " Oct 30 20:53:58.649632 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.649513 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-model-cache" pod "b2b57028-ba25-496d-834e-5433b95cc2a4" (UID: "b2b57028-ba25-496d-834e-5433b95cc2a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:53:58.649632 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.649543 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-tmp-dir\") pod \"b2b57028-ba25-496d-834e-5433b95cc2a4\" (UID: \"b2b57028-ba25-496d-834e-5433b95cc2a4\") " Oct 30 20:53:58.649632 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.649616 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-home" pod "b2b57028-ba25-496d-834e-5433b95cc2a4" (UID: "b2b57028-ba25-496d-834e-5433b95cc2a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:53:58.649790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.649746 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-tmp-dir" pod "b2b57028-ba25-496d-834e-5433b95cc2a4" (UID: "b2b57028-ba25-496d-834e-5433b95cc2a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:53:58.650371 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.650339 5755 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-model-cache\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:58.650560 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.650375 5755 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-home\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:58.650560 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.650408 5755 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-tmp-dir\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:58.652028 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.652002 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b57028-ba25-496d-834e-5433b95cc2a4-kube-api-access-gbzdx" pod "b2b57028-ba25-496d-834e-5433b95cc2a4" (UID: "b2b57028-ba25-496d-834e-5433b95cc2a4"). InnerVolumeSpecName "kube-api-access-gbzdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:53:58.652512 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.652488 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-dshm" pod "b2b57028-ba25-496d-834e-5433b95cc2a4" (UID: "b2b57028-ba25-496d-834e-5433b95cc2a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:53:58.652631 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.652492 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b57028-ba25-496d-834e-5433b95cc2a4-tls-certs" pod "b2b57028-ba25-496d-834e-5433b95cc2a4" (UID: "b2b57028-ba25-496d-834e-5433b95cc2a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:58.751726 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.751696 5755 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b57028-ba25-496d-834e-5433b95cc2a4-dshm\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:58.751726 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.751725 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbzdx\" (UniqueName: \"kubernetes.io/projected/b2b57028-ba25-496d-834e-5433b95cc2a4-kube-api-access-gbzdx\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:58.751921 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:58.751736 5755 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b57028-ba25-496d-834e-5433b95cc2a4-tls-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:59.383214 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:59.383180 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" Oct 30 20:53:59.383690 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:59.383215 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" event={"ID":"b2b57028-ba25-496d-834e-5433b95cc2a4","Type":"ContainerDied","Data":"5d3db2d24539b5d1a510588a49f9aca17cef7e5479c9f817f1440cf718647bbd"} Oct 30 20:53:59.383690 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:59.383260 5755 scope.go:122] "RemoveContainer" containerID="48d4df7a0c35b7ede7363d91a660bbe248703e5ccc553ae7a3c39ea0361c9749" Oct 30 20:53:59.457654 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:59.454880 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:59.576143 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:59.560420 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64"] Oct 30 20:53:59.576143 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:53:59.570724 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b32941/distinct-w0-b32941-kserve-667ffcb56d-5dn64"] Oct 30 20:54:01.213088 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:01.213043 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" path="/var/lib/kubelet/pods/b2b57028-ba25-496d-834e-5433b95cc2a4/volumes" Oct 30 20:54:02.074352 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:02.074319 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766"] Oct 30 20:54:02.074572 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:02.074535 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" podUID="4dc87293-4490-43f3-851a-05f393007b1a" containerName="istio-proxy" containerID="cri-o://802d17a33c095d0f01e15200d8ff2f62e5a27b46c7c4f2dc6c5c30d6039295a5" gracePeriod=30 Oct 30 20:54:02.907364 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:02.907326 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22"] Oct 30 20:54:02.907777 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:02.907580 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" podUID="00f11563-7c2c-4c14-a6a9-1b2f751a057d" containerName="istio-proxy" containerID="cri-o://4c56332dd37ad287e96900b62b4e08f8455a1ea218cd9516cbbe4ff472659aeb" gracePeriod=30 Oct 30 20:54:07.373590 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.373568 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:54:07.446057 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.446024 5755 generic.go:356] "Generic (PLEG): container finished" podID="4dc87293-4490-43f3-851a-05f393007b1a" containerID="802d17a33c095d0f01e15200d8ff2f62e5a27b46c7c4f2dc6c5c30d6039295a5" exitCode=0 Oct 30 20:54:07.446210 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.446071 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" Oct 30 20:54:07.446210 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.446103 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" event={"ID":"4dc87293-4490-43f3-851a-05f393007b1a","Type":"ContainerDied","Data":"802d17a33c095d0f01e15200d8ff2f62e5a27b46c7c4f2dc6c5c30d6039295a5"} Oct 30 20:54:07.446210 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.446157 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766" event={"ID":"4dc87293-4490-43f3-851a-05f393007b1a","Type":"ContainerDied","Data":"30cce890b2fa9850ae93ae9be14dbb3c0ea8db9047a76c43071aa8fcc133584b"} Oct 30 20:54:07.446210 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.446179 5755 scope.go:122] "RemoveContainer" containerID="802d17a33c095d0f01e15200d8ff2f62e5a27b46c7c4f2dc6c5c30d6039295a5" Oct 30 20:54:07.456849 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.456820 5755 scope.go:122] "RemoveContainer" containerID="802d17a33c095d0f01e15200d8ff2f62e5a27b46c7c4f2dc6c5c30d6039295a5" Oct 30 20:54:07.457122 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:54:07.457101 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802d17a33c095d0f01e15200d8ff2f62e5a27b46c7c4f2dc6c5c30d6039295a5\": container with ID starting with 802d17a33c095d0f01e15200d8ff2f62e5a27b46c7c4f2dc6c5c30d6039295a5 not found: ID does not exist" containerID="802d17a33c095d0f01e15200d8ff2f62e5a27b46c7c4f2dc6c5c30d6039295a5" Oct 30 20:54:07.457169 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.457132 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802d17a33c095d0f01e15200d8ff2f62e5a27b46c7c4f2dc6c5c30d6039295a5"} err="failed to get container status \"802d17a33c095d0f01e15200d8ff2f62e5a27b46c7c4f2dc6c5c30d6039295a5\": rpc error: code = NotFound desc = could not find container \"802d17a33c095d0f01e15200d8ff2f62e5a27b46c7c4f2dc6c5c30d6039295a5\": container with ID starting with 802d17a33c095d0f01e15200d8ff2f62e5a27b46c7c4f2dc6c5c30d6039295a5 not found: ID does not exist" Oct 30 20:54:07.492380 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.492355 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-credential-socket\") pod \"4dc87293-4490-43f3-851a-05f393007b1a\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " Oct 30 20:54:07.492517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.492422 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4dc87293-4490-43f3-851a-05f393007b1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc87293-4490-43f3-851a-05f393007b1a-istio-podinfo\") pod \"4dc87293-4490-43f3-851a-05f393007b1a\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " Oct 30 20:54:07.492517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.492482 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-istio-data\") pod \"4dc87293-4490-43f3-851a-05f393007b1a\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " Oct 30 20:54:07.492629 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.492589 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-credential-socket" pod "4dc87293-4490-43f3-851a-05f393007b1a" (UID: "4dc87293-4490-43f3-851a-05f393007b1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:07.492629 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.492597 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-workload-socket\") pod \"4dc87293-4490-43f3-851a-05f393007b1a\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " Oct 30 20:54:07.492728 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.492627 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dc87293-4490-43f3-851a-05f393007b1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc87293-4490-43f3-851a-05f393007b1a-istiod-ca-cert\") pod \"4dc87293-4490-43f3-851a-05f393007b1a\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " Oct 30 20:54:07.492728 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.492667 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dc87293-4490-43f3-851a-05f393007b1a-kube-api-access-q7g88\" (UniqueName: \"kubernetes.io/projected/4dc87293-4490-43f3-851a-05f393007b1a-kube-api-access-q7g88\") pod \"4dc87293-4490-43f3-851a-05f393007b1a\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " Oct 30 20:54:07.492828 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.492796 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-workload-certs\") pod \"4dc87293-4490-43f3-851a-05f393007b1a\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " Oct 30 20:54:07.492828 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.492786 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-istio-data" pod "4dc87293-4490-43f3-851a-05f393007b1a" (UID: "4dc87293-4490-43f3-851a-05f393007b1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:07.492931 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.492867 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-workload-socket" pod "4dc87293-4490-43f3-851a-05f393007b1a" (UID: "4dc87293-4490-43f3-851a-05f393007b1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:07.492931 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.492917 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dc87293-4490-43f3-851a-05f393007b1a-istio-token\" (UniqueName: \"kubernetes.io/projected/4dc87293-4490-43f3-851a-05f393007b1a-istio-token\") pod \"4dc87293-4490-43f3-851a-05f393007b1a\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " Oct 30 20:54:07.493025 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.492964 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc87293-4490-43f3-851a-05f393007b1a-istiod-ca-cert" pod "4dc87293-4490-43f3-851a-05f393007b1a" (UID: "4dc87293-4490-43f3-851a-05f393007b1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:07.493025 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.492990 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-istio-envoy\") pod \"4dc87293-4490-43f3-851a-05f393007b1a\" (UID: \"4dc87293-4490-43f3-851a-05f393007b1a\") " Oct 30 20:54:07.493123 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.493062 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-workload-certs" pod "4dc87293-4490-43f3-851a-05f393007b1a" (UID: "4dc87293-4490-43f3-851a-05f393007b1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:07.493568 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.493545 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:07.493691 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.493571 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:07.493691 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.493586 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:07.493691 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.493599 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:07.493691 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.493612 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc87293-4490-43f3-851a-05f393007b1a-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:07.494866 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.494844 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dc87293-4490-43f3-851a-05f393007b1a-istio-podinfo" pod "4dc87293-4490-43f3-851a-05f393007b1a" (UID: "4dc87293-4490-43f3-851a-05f393007b1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:54:07.495023 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.495001 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc87293-4490-43f3-851a-05f393007b1a-kube-api-access-q7g88" pod "4dc87293-4490-43f3-851a-05f393007b1a" (UID: "4dc87293-4490-43f3-851a-05f393007b1a"). InnerVolumeSpecName "kube-api-access-q7g88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:07.495424 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.495379 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc87293-4490-43f3-851a-05f393007b1a-istio-token" pod "4dc87293-4490-43f3-851a-05f393007b1a" (UID: "4dc87293-4490-43f3-851a-05f393007b1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:07.495633 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.495614 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-istio-envoy" pod "4dc87293-4490-43f3-851a-05f393007b1a" (UID: "4dc87293-4490-43f3-851a-05f393007b1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:07.595717 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.595634 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7g88\" (UniqueName: \"kubernetes.io/projected/4dc87293-4490-43f3-851a-05f393007b1a-kube-api-access-q7g88\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:07.595717 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.595672 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc87293-4490-43f3-851a-05f393007b1a-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:07.595717 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.595687 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc87293-4490-43f3-851a-05f393007b1a-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:07.595717 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.595702 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc87293-4490-43f3-851a-05f393007b1a-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:07.885524 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.883768 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766"] Oct 30 20:54:07.893547 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:07.893505 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-vh766"] Oct 30 20:54:08.169588 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.169562 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:54:08.209030 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.209004 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-token\" (UniqueName: \"kubernetes.io/projected/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-token\") pod \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " Oct 30 20:54:08.209176 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.209050 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-workload-socket\") pod \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " Oct 30 20:54:08.209176 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.209082 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-envoy\") pod \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " Oct 30 20:54:08.209176 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.209120 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f11563-7c2c-4c14-a6a9-1b2f751a057d-kube-api-access-48jhc\" (UniqueName: \"kubernetes.io/projected/00f11563-7c2c-4c14-a6a9-1b2f751a057d-kube-api-access-48jhc\") pod \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " Oct 30 20:54:08.209327 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.209176 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istiod-ca-cert\") pod \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " Oct 30 20:54:08.209327 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.209237 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-workload-certs\") pod \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " Oct 30 20:54:08.209327 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.209287 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-data\") pod \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " Oct 30 20:54:08.209527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.209350 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-podinfo\") pod \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " Oct 30 20:54:08.209527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.209356 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-workload-socket" pod "00f11563-7c2c-4c14-a6a9-1b2f751a057d" (UID: "00f11563-7c2c-4c14-a6a9-1b2f751a057d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:08.209527 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.209439 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-credential-socket\") pod \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\" (UID: \"00f11563-7c2c-4c14-a6a9-1b2f751a057d\") " Oct 30 20:54:08.209670 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.209555 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istiod-ca-cert" pod "00f11563-7c2c-4c14-a6a9-1b2f751a057d" (UID: "00f11563-7c2c-4c14-a6a9-1b2f751a057d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:08.209670 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.209588 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-workload-certs" pod "00f11563-7c2c-4c14-a6a9-1b2f751a057d" (UID: "00f11563-7c2c-4c14-a6a9-1b2f751a057d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:08.209862 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.209838 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-data" pod "00f11563-7c2c-4c14-a6a9-1b2f751a057d" (UID: "00f11563-7c2c-4c14-a6a9-1b2f751a057d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:08.210089 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.209915 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-credential-socket" pod "00f11563-7c2c-4c14-a6a9-1b2f751a057d" (UID: "00f11563-7c2c-4c14-a6a9-1b2f751a057d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:08.210335 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.210312 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:08.210442 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.210343 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:08.210442 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.210358 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:08.210442 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.210373 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:08.210442 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.210425 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:08.211495 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.211475 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-envoy" pod "00f11563-7c2c-4c14-a6a9-1b2f751a057d" (UID: "00f11563-7c2c-4c14-a6a9-1b2f751a057d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:08.211592 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.211548 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f11563-7c2c-4c14-a6a9-1b2f751a057d-kube-api-access-48jhc" pod "00f11563-7c2c-4c14-a6a9-1b2f751a057d" (UID: "00f11563-7c2c-4c14-a6a9-1b2f751a057d"). InnerVolumeSpecName "kube-api-access-48jhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:08.211878 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.211862 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-token" pod "00f11563-7c2c-4c14-a6a9-1b2f751a057d" (UID: "00f11563-7c2c-4c14-a6a9-1b2f751a057d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:08.211952 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.211928 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-podinfo" pod "00f11563-7c2c-4c14-a6a9-1b2f751a057d" (UID: "00f11563-7c2c-4c14-a6a9-1b2f751a057d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:54:08.312039 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.312003 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:08.312039 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.312041 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48jhc\" (UniqueName: \"kubernetes.io/projected/00f11563-7c2c-4c14-a6a9-1b2f751a057d-kube-api-access-48jhc\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:08.312290 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.312056 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:08.312290 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.312070 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f11563-7c2c-4c14-a6a9-1b2f751a057d-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:08.458214 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.458133 5755 generic.go:356] "Generic (PLEG): container finished" podID="00f11563-7c2c-4c14-a6a9-1b2f751a057d" containerID="4c56332dd37ad287e96900b62b4e08f8455a1ea218cd9516cbbe4ff472659aeb" exitCode=0 Oct 30 20:54:08.458214 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.458182 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" Oct 30 20:54:08.458760 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.458215 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" event={"ID":"00f11563-7c2c-4c14-a6a9-1b2f751a057d","Type":"ContainerDied","Data":"4c56332dd37ad287e96900b62b4e08f8455a1ea218cd9516cbbe4ff472659aeb"} Oct 30 20:54:08.458760 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.458280 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22" event={"ID":"00f11563-7c2c-4c14-a6a9-1b2f751a057d","Type":"ContainerDied","Data":"0e836cd62067a2e868404d8332ff863a991e9754dac164958d12fb161fee8efb"} Oct 30 20:54:08.458760 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.458315 5755 scope.go:122] "RemoveContainer" containerID="4c56332dd37ad287e96900b62b4e08f8455a1ea218cd9516cbbe4ff472659aeb" Oct 30 20:54:08.470369 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.470350 5755 scope.go:122] "RemoveContainer" containerID="4c56332dd37ad287e96900b62b4e08f8455a1ea218cd9516cbbe4ff472659aeb" Oct 30 20:54:08.470676 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:54:08.470656 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c56332dd37ad287e96900b62b4e08f8455a1ea218cd9516cbbe4ff472659aeb\": container with ID starting with 4c56332dd37ad287e96900b62b4e08f8455a1ea218cd9516cbbe4ff472659aeb not found: ID does not exist" containerID="4c56332dd37ad287e96900b62b4e08f8455a1ea218cd9516cbbe4ff472659aeb" Oct 30 20:54:08.470724 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.470687 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c56332dd37ad287e96900b62b4e08f8455a1ea218cd9516cbbe4ff472659aeb"} err="failed to get container status \"4c56332dd37ad287e96900b62b4e08f8455a1ea218cd9516cbbe4ff472659aeb\": rpc error: code = NotFound desc = could not find container \"4c56332dd37ad287e96900b62b4e08f8455a1ea218cd9516cbbe4ff472659aeb\": container with ID starting with 4c56332dd37ad287e96900b62b4e08f8455a1ea218cd9516cbbe4ff472659aeb not found: ID does not exist" Oct 30 20:54:08.628854 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.628821 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22"] Oct 30 20:54:08.637102 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:08.637066 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-rvg22"] Oct 30 20:54:09.213759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:09.213726 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f11563-7c2c-4c14-a6a9-1b2f751a057d" path="/var/lib/kubelet/pods/00f11563-7c2c-4c14-a6a9-1b2f751a057d/volumes" Oct 30 20:54:09.214430 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:09.214407 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dc87293-4490-43f3-851a-05f393007b1a" path="/var/lib/kubelet/pods/4dc87293-4490-43f3-851a-05f393007b1a/volumes" Oct 30 20:54:11.256328 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:54:11.256283 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:54:17.360065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:17.360030 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j"] Oct 30 20:54:17.360530 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:17.360277 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" podUID="1ddd9841-3766-4d6a-984b-1410f58eba31" containerName="istio-proxy" containerID="cri-o://b7cb72ef22895d1aed2ed05360b16e18d66f8256b4dd51cd16e45cf76b338b1a" gracePeriod=30 Oct 30 20:54:17.818254 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:17.818218 5755 scope.go:122] "RemoveContainer" containerID="3efbe06a3903c75e46c35a6469aeee0d6590f1159ebbf612a2184b3add133fd0" Oct 30 20:54:17.829574 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:17.829538 5755 scope.go:122] "RemoveContainer" containerID="e45d09601735e14c9a263a097b63c9d023b2ef5d47753a5c9d0518b468864706" Oct 30 20:54:17.840783 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:17.840756 5755 scope.go:122] "RemoveContainer" containerID="6bc07c3195459306cfb707f6c9835a4524f435e3d66d64530377202c646854e7" Oct 30 20:54:17.854948 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:17.854911 5755 scope.go:122] "RemoveContainer" containerID="589bec91b575f833c2fb11638354dbc85682aeccf0fa76047dba9f043a90c218" Oct 30 20:54:17.864505 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:17.864486 5755 scope.go:122] "RemoveContainer" containerID="809412ae3c8d102a1f5953574466bcdf2bd3682d2b8d49e89474fb397a32a9e9" Oct 30 20:54:17.875607 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:17.875588 5755 scope.go:122] "RemoveContainer" containerID="44ff3a6825eb266d467996641d7d79df2bdd5daef53f7e637f456025e5635b7c" Oct 30 20:54:17.884359 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:17.884259 5755 scope.go:122] "RemoveContainer" containerID="ff154c042bdfb047ac2a11e6fd4f8292d13162524a7f82cb70440cad05733942" Oct 30 20:54:19.067607 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.067576 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2"] Oct 30 20:54:19.068146 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.068132 5755 cpu_manager.go:414] "containerMap: removing container" podUID="00f11563-7c2c-4c14-a6a9-1b2f751a057d" containerName="istio-proxy" Oct 30 20:54:19.068189 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.068147 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f11563-7c2c-4c14-a6a9-1b2f751a057d" containerName="istio-proxy" Oct 30 20:54:19.068189 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.068179 5755 cpu_manager.go:414] "containerMap: removing container" podUID="4dc87293-4490-43f3-851a-05f393007b1a" containerName="istio-proxy" Oct 30 20:54:19.068189 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.068186 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dc87293-4490-43f3-851a-05f393007b1a" containerName="istio-proxy" Oct 30 20:54:19.068282 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.068195 5755 cpu_manager.go:414] "containerMap: removing container" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" Oct 30 20:54:19.068282 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.068200 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" Oct 30 20:54:19.068282 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.068209 5755 cpu_manager.go:414] "containerMap: removing container" podUID="db450192-3182-4d15-9af2-cb9875996024" containerName="revoke-keys" Oct 30 20:54:19.068282 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.068213 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db450192-3182-4d15-9af2-cb9875996024" containerName="revoke-keys" Oct 30 20:54:19.068416 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.068307 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dc87293-4490-43f3-851a-05f393007b1a" containerName="istio-proxy" Oct 30 20:54:19.068416 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.068315 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b57028-ba25-496d-834e-5433b95cc2a4" containerName="main" Oct 30 20:54:19.068416 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.068323 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="db450192-3182-4d15-9af2-cb9875996024" containerName="revoke-keys" Oct 30 20:54:19.068416 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.068333 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f11563-7c2c-4c14-a6a9-1b2f751a057d" containerName="istio-proxy" Oct 30 20:54:19.073679 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.073663 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" Oct 30 20:54:19.141306 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.141273 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2"] Oct 30 20:54:19.180450 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.180417 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:54:19.180608 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.180479 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:54:19.187721 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.187691 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383f66e1-rxkm2\" (UID: \"fac7b7aa-6821-4ea6-bbc7-fd0990b98161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" Oct 30 20:54:19.187985 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.187967 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383f66e1-rxkm2\" (UID: \"fac7b7aa-6821-4ea6-bbc7-fd0990b98161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" Oct 30 20:54:19.188124 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.188105 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xqp4\" (UniqueName: \"kubernetes.io/projected/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-kube-api-access-2xqp4\") pod \"maas-api-revoke-sub-383f66e1-rxkm2\" (UID: \"fac7b7aa-6821-4ea6-bbc7-fd0990b98161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" Oct 30 20:54:19.289998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.289967 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383f66e1-rxkm2\" (UID: \"fac7b7aa-6821-4ea6-bbc7-fd0990b98161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" Oct 30 20:54:19.290186 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.290035 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xqp4\" (UniqueName: \"kubernetes.io/projected/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-kube-api-access-2xqp4\") pod \"maas-api-revoke-sub-383f66e1-rxkm2\" (UID: \"fac7b7aa-6821-4ea6-bbc7-fd0990b98161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" Oct 30 20:54:19.290186 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.290086 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383f66e1-rxkm2\" (UID: \"fac7b7aa-6821-4ea6-bbc7-fd0990b98161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" Oct 30 20:54:19.290730 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.290695 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383f66e1-rxkm2\" (UID: \"fac7b7aa-6821-4ea6-bbc7-fd0990b98161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" Oct 30 20:54:19.292765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.292749 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383f66e1-rxkm2\" (UID: \"fac7b7aa-6821-4ea6-bbc7-fd0990b98161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" Oct 30 20:54:19.350025 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.349956 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xqp4\" (UniqueName: \"kubernetes.io/projected/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-kube-api-access-2xqp4\") pod \"maas-api-revoke-sub-383f66e1-rxkm2\" (UID: \"fac7b7aa-6821-4ea6-bbc7-fd0990b98161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" Oct 30 20:54:19.386829 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.386796 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" Oct 30 20:54:19.826366 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:54:19.826331 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac7b7aa_6821_4ea6_bbc7_fd0990b98161.slice/crio-26c2c9a00bfd935ca233a67ad89827396b41be147f72b2885ad91a37eacd0189 WatchSource:0}: Error finding container 26c2c9a00bfd935ca233a67ad89827396b41be147f72b2885ad91a37eacd0189: Status 404 returned error can't find the container with id 26c2c9a00bfd935ca233a67ad89827396b41be147f72b2885ad91a37eacd0189 Oct 30 20:54:19.827107 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:19.827080 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2"] Oct 30 20:54:20.638843 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:20.638648 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" event={"ID":"fac7b7aa-6821-4ea6-bbc7-fd0990b98161","Type":"ContainerStarted","Data":"20b3dce11b28b2c6c34d0a825f7c494a4e69a2d2fac49ab4ee5efad791411ebb"} Oct 30 20:54:20.638843 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:20.638753 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" event={"ID":"fac7b7aa-6821-4ea6-bbc7-fd0990b98161","Type":"ContainerStarted","Data":"26c2c9a00bfd935ca233a67ad89827396b41be147f72b2885ad91a37eacd0189"} Oct 30 20:54:20.792932 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:20.792872 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" podStartSLOduration=1.792851564 podStartE2EDuration="1.792851564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:20.78999914 +0000 UTC m=+1623.716556072" watchObservedRunningTime="2026-10-30 20:54:20.792851564 +0000 UTC m=+1623.719408480" Oct 30 20:54:21.657945 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:21.657892 5755 generic.go:356] "Generic (PLEG): container finished" podID="fac7b7aa-6821-4ea6-bbc7-fd0990b98161" containerID="20b3dce11b28b2c6c34d0a825f7c494a4e69a2d2fac49ab4ee5efad791411ebb" exitCode=0 Oct 30 20:54:21.658539 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:21.657949 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" event={"ID":"fac7b7aa-6821-4ea6-bbc7-fd0990b98161","Type":"ContainerDied","Data":"20b3dce11b28b2c6c34d0a825f7c494a4e69a2d2fac49ab4ee5efad791411ebb"} Oct 30 20:54:22.675327 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.675231 5755 generic.go:356] "Generic (PLEG): container finished" podID="1ddd9841-3766-4d6a-984b-1410f58eba31" containerID="b7cb72ef22895d1aed2ed05360b16e18d66f8256b4dd51cd16e45cf76b338b1a" exitCode=0 Oct 30 20:54:22.675327 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.675303 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" event={"ID":"1ddd9841-3766-4d6a-984b-1410f58eba31","Type":"ContainerDied","Data":"b7cb72ef22895d1aed2ed05360b16e18d66f8256b4dd51cd16e45cf76b338b1a"} Oct 30 20:54:22.711428 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.711362 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:54:22.789484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.788893 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-workload-socket\") pod \"1ddd9841-3766-4d6a-984b-1410f58eba31\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " Oct 30 20:54:22.789484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.788972 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-token\" (UniqueName: \"kubernetes.io/projected/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-token\") pod \"1ddd9841-3766-4d6a-984b-1410f58eba31\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " Oct 30 20:54:22.789484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.789028 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-podinfo\") pod \"1ddd9841-3766-4d6a-984b-1410f58eba31\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " Oct 30 20:54:22.789484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.789059 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-data\") pod \"1ddd9841-3766-4d6a-984b-1410f58eba31\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " Oct 30 20:54:22.789484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.789092 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-envoy\") pod \"1ddd9841-3766-4d6a-984b-1410f58eba31\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " Oct 30 20:54:22.789484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.789143 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-workload-certs\") pod \"1ddd9841-3766-4d6a-984b-1410f58eba31\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " Oct 30 20:54:22.789484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.789283 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ddd9841-3766-4d6a-984b-1410f58eba31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ddd9841-3766-4d6a-984b-1410f58eba31-istiod-ca-cert\") pod \"1ddd9841-3766-4d6a-984b-1410f58eba31\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " Oct 30 20:54:22.789484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.789427 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ddd9841-3766-4d6a-984b-1410f58eba31-kube-api-access-zx25m\" (UniqueName: \"kubernetes.io/projected/1ddd9841-3766-4d6a-984b-1410f58eba31-kube-api-access-zx25m\") pod \"1ddd9841-3766-4d6a-984b-1410f58eba31\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " Oct 30 20:54:22.790072 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.789497 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-credential-socket\") pod \"1ddd9841-3766-4d6a-984b-1410f58eba31\" (UID: \"1ddd9841-3766-4d6a-984b-1410f58eba31\") " Oct 30 20:54:22.791343 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.791283 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ddd9841-3766-4d6a-984b-1410f58eba31-istiod-ca-cert" pod "1ddd9841-3766-4d6a-984b-1410f58eba31" (UID: "1ddd9841-3766-4d6a-984b-1410f58eba31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:22.791766 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.791740 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-workload-certs" pod "1ddd9841-3766-4d6a-984b-1410f58eba31" (UID: "1ddd9841-3766-4d6a-984b-1410f58eba31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:22.792531 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.792475 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-workload-socket" pod "1ddd9841-3766-4d6a-984b-1410f58eba31" (UID: "1ddd9841-3766-4d6a-984b-1410f58eba31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:22.803480 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.799716 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-data" pod "1ddd9841-3766-4d6a-984b-1410f58eba31" (UID: "1ddd9841-3766-4d6a-984b-1410f58eba31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:22.803480 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.803269 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-credential-socket" pod "1ddd9841-3766-4d6a-984b-1410f58eba31" (UID: "1ddd9841-3766-4d6a-984b-1410f58eba31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:22.815338 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.814920 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-token" pod "1ddd9841-3766-4d6a-984b-1410f58eba31" (UID: "1ddd9841-3766-4d6a-984b-1410f58eba31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:22.821520 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.820556 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-envoy" pod "1ddd9841-3766-4d6a-984b-1410f58eba31" (UID: "1ddd9841-3766-4d6a-984b-1410f58eba31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:22.821520 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.821025 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-podinfo" pod "1ddd9841-3766-4d6a-984b-1410f58eba31" (UID: "1ddd9841-3766-4d6a-984b-1410f58eba31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:54:22.831549 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.831145 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ddd9841-3766-4d6a-984b-1410f58eba31-kube-api-access-zx25m" pod "1ddd9841-3766-4d6a-984b-1410f58eba31" (UID: "1ddd9841-3766-4d6a-984b-1410f58eba31"). InnerVolumeSpecName "kube-api-access-zx25m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:22.915510 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.909638 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx25m\" (UniqueName: \"kubernetes.io/projected/1ddd9841-3766-4d6a-984b-1410f58eba31-kube-api-access-zx25m\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:22.915510 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.909672 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:22.915510 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.909688 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:22.915510 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.909704 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:22.915510 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.909718 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:22.915510 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.909733 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:22.915510 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.909747 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:22.915510 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.909761 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ddd9841-3766-4d6a-984b-1410f58eba31-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:22.915510 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.909774 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ddd9841-3766-4d6a-984b-1410f58eba31-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:22.938752 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:22.938626 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" Oct 30 20:54:23.010617 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.010583 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-maas-api-openshift-ca\") pod \"fac7b7aa-6821-4ea6-bbc7-fd0990b98161\" (UID: \"fac7b7aa-6821-4ea6-bbc7-fd0990b98161\") " Oct 30 20:54:23.010817 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.010666 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-kube-api-access-2xqp4\" (UniqueName: \"kubernetes.io/projected/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-kube-api-access-2xqp4\") pod \"fac7b7aa-6821-4ea6-bbc7-fd0990b98161\" (UID: \"fac7b7aa-6821-4ea6-bbc7-fd0990b98161\") " Oct 30 20:54:23.010890 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.010870 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-maas-api-xks-ca\") pod \"fac7b7aa-6821-4ea6-bbc7-fd0990b98161\" (UID: \"fac7b7aa-6821-4ea6-bbc7-fd0990b98161\") " Oct 30 20:54:23.011008 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.010977 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-maas-api-openshift-ca" pod "fac7b7aa-6821-4ea6-bbc7-fd0990b98161" (UID: "fac7b7aa-6821-4ea6-bbc7-fd0990b98161"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:23.011419 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.011365 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:23.013002 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.012982 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-maas-api-xks-ca" pod "fac7b7aa-6821-4ea6-bbc7-fd0990b98161" (UID: "fac7b7aa-6821-4ea6-bbc7-fd0990b98161"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:23.013331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.013304 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-kube-api-access-2xqp4" pod "fac7b7aa-6821-4ea6-bbc7-fd0990b98161" (UID: "fac7b7aa-6821-4ea6-bbc7-fd0990b98161"). InnerVolumeSpecName "kube-api-access-2xqp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:23.142413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.141634 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xqp4\" (UniqueName: \"kubernetes.io/projected/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-kube-api-access-2xqp4\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:23.142413 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.141673 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fac7b7aa-6821-4ea6-bbc7-fd0990b98161-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:23.684200 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.684110 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" Oct 30 20:54:23.684200 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.684136 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j" event={"ID":"1ddd9841-3766-4d6a-984b-1410f58eba31","Type":"ContainerDied","Data":"d905b990f0a8d76bdd1f411c7c50066d2ead4557760c7a9f3960cd46602956ea"} Oct 30 20:54:23.684753 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.684207 5755 scope.go:122] "RemoveContainer" containerID="b7cb72ef22895d1aed2ed05360b16e18d66f8256b4dd51cd16e45cf76b338b1a" Oct 30 20:54:23.686009 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.685984 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" event={"ID":"fac7b7aa-6821-4ea6-bbc7-fd0990b98161","Type":"ContainerDied","Data":"26c2c9a00bfd935ca233a67ad89827396b41be147f72b2885ad91a37eacd0189"} Oct 30 20:54:23.686113 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.686032 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26c2c9a00bfd935ca233a67ad89827396b41be147f72b2885ad91a37eacd0189" Oct 30 20:54:23.686160 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.686127 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2" Oct 30 20:54:23.909431 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.909376 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j"] Oct 30 20:54:23.914556 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:23.914523 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8b54j"] Oct 30 20:54:25.212910 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:25.212877 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ddd9841-3766-4d6a-984b-1410f58eba31" path="/var/lib/kubelet/pods/1ddd9841-3766-4d6a-984b-1410f58eba31/volumes" Oct 30 20:54:38.600145 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:38.600069 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk"] Oct 30 20:54:38.648283 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:38.648254 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0ff4fc-4wgxk"] Oct 30 20:54:39.213400 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:39.213357 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db450192-3182-4d15-9af2-cb9875996024" path="/var/lib/kubelet/pods/db450192-3182-4d15-9af2-cb9875996024/volumes" Oct 30 20:54:41.257098 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:54:41.257063 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:54:44.325663 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.325621 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw"] Oct 30 20:54:44.326414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.326374 5755 cpu_manager.go:414] "containerMap: removing container" podUID="1ddd9841-3766-4d6a-984b-1410f58eba31" containerName="istio-proxy" Oct 30 20:54:44.326466 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.326416 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ddd9841-3766-4d6a-984b-1410f58eba31" containerName="istio-proxy" Oct 30 20:54:44.326466 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.326448 5755 cpu_manager.go:414] "containerMap: removing container" podUID="fac7b7aa-6821-4ea6-bbc7-fd0990b98161" containerName="revoke-keys" Oct 30 20:54:44.326466 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.326453 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac7b7aa-6821-4ea6-bbc7-fd0990b98161" containerName="revoke-keys" Oct 30 20:54:44.326560 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.326555 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ddd9841-3766-4d6a-984b-1410f58eba31" containerName="istio-proxy" Oct 30 20:54:44.326596 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.326566 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac7b7aa-6821-4ea6-bbc7-fd0990b98161" containerName="revoke-keys" Oct 30 20:54:44.330180 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.330163 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" Oct 30 20:54:44.392369 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.392331 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw"] Oct 30 20:54:44.426736 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.426702 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dmfn\" (UniqueName: \"kubernetes.io/projected/7fd7656a-026c-4b86-a016-9e11446d27ae-kube-api-access-5dmfn\") pod \"maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw\" (UID: \"7fd7656a-026c-4b86-a016-9e11446d27ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" Oct 30 20:54:44.426896 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.426763 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd7656a-026c-4b86-a016-9e11446d27ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw\" (UID: \"7fd7656a-026c-4b86-a016-9e11446d27ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" Oct 30 20:54:44.427100 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.427075 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd7656a-026c-4b86-a016-9e11446d27ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw\" (UID: \"7fd7656a-026c-4b86-a016-9e11446d27ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" Oct 30 20:54:44.437495 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.437465 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:54:44.437648 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.437533 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:54:44.529007 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.528972 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd7656a-026c-4b86-a016-9e11446d27ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw\" (UID: \"7fd7656a-026c-4b86-a016-9e11446d27ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" Oct 30 20:54:44.529215 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.529054 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dmfn\" (UniqueName: \"kubernetes.io/projected/7fd7656a-026c-4b86-a016-9e11446d27ae-kube-api-access-5dmfn\") pod \"maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw\" (UID: \"7fd7656a-026c-4b86-a016-9e11446d27ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" Oct 30 20:54:44.529215 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.529096 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd7656a-026c-4b86-a016-9e11446d27ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw\" (UID: \"7fd7656a-026c-4b86-a016-9e11446d27ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" Oct 30 20:54:44.529731 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.529706 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd7656a-026c-4b86-a016-9e11446d27ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw\" (UID: \"7fd7656a-026c-4b86-a016-9e11446d27ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" Oct 30 20:54:44.531663 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.531641 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd7656a-026c-4b86-a016-9e11446d27ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw\" (UID: \"7fd7656a-026c-4b86-a016-9e11446d27ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" Oct 30 20:54:44.588971 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.588898 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dmfn\" (UniqueName: \"kubernetes.io/projected/7fd7656a-026c-4b86-a016-9e11446d27ae-kube-api-access-5dmfn\") pod \"maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw\" (UID: \"7fd7656a-026c-4b86-a016-9e11446d27ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" Oct 30 20:54:44.644718 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:44.644683 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" Oct 30 20:54:45.025852 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:45.025821 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw"] Oct 30 20:54:45.026450 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:54:45.026423 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd7656a_026c_4b86_a016_9e11446d27ae.slice/crio-a53c98c6718529109d2fc40763c362ef9b0728c3bd2f41cd83117c05cb22ae96 WatchSource:0}: Error finding container a53c98c6718529109d2fc40763c362ef9b0728c3bd2f41cd83117c05cb22ae96: Status 404 returned error can't find the container with id a53c98c6718529109d2fc40763c362ef9b0728c3bd2f41cd83117c05cb22ae96 Oct 30 20:54:45.870854 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:45.870810 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" event={"ID":"7fd7656a-026c-4b86-a016-9e11446d27ae","Type":"ContainerStarted","Data":"cae441a5ef6bee1b07345280a943e9118802ec614dc050bbe5c287c3500c926f"} Oct 30 20:54:45.871405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:45.870886 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" event={"ID":"7fd7656a-026c-4b86-a016-9e11446d27ae","Type":"ContainerStarted","Data":"a53c98c6718529109d2fc40763c362ef9b0728c3bd2f41cd83117c05cb22ae96"} Oct 30 20:54:45.995269 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:45.995200 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" podStartSLOduration=1.9951793759999998 podStartE2EDuration="1.995179376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:45.992122641 +0000 UTC m=+1648.918679580" watchObservedRunningTime="2026-10-30 20:54:45.995179376 +0000 UTC m=+1648.921736293" Oct 30 20:54:46.929583 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:46.929536 5755 generic.go:356] "Generic (PLEG): container finished" podID="7fd7656a-026c-4b86-a016-9e11446d27ae" containerID="cae441a5ef6bee1b07345280a943e9118802ec614dc050bbe5c287c3500c926f" exitCode=0 Oct 30 20:54:46.930116 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:46.929648 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" event={"ID":"7fd7656a-026c-4b86-a016-9e11446d27ae","Type":"ContainerDied","Data":"cae441a5ef6bee1b07345280a943e9118802ec614dc050bbe5c287c3500c926f"} Oct 30 20:54:48.130344 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:48.130311 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" Oct 30 20:54:48.290673 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:48.290639 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd7656a-026c-4b86-a016-9e11446d27ae-kube-api-access-5dmfn\" (UniqueName: \"kubernetes.io/projected/7fd7656a-026c-4b86-a016-9e11446d27ae-kube-api-access-5dmfn\") pod \"7fd7656a-026c-4b86-a016-9e11446d27ae\" (UID: \"7fd7656a-026c-4b86-a016-9e11446d27ae\") " Oct 30 20:54:48.290673 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:48.290692 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fd7656a-026c-4b86-a016-9e11446d27ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd7656a-026c-4b86-a016-9e11446d27ae-maas-api-openshift-ca\") pod \"7fd7656a-026c-4b86-a016-9e11446d27ae\" (UID: \"7fd7656a-026c-4b86-a016-9e11446d27ae\") " Oct 30 20:54:48.291055 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:48.291031 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fd7656a-026c-4b86-a016-9e11446d27ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd7656a-026c-4b86-a016-9e11446d27ae-maas-api-xks-ca\") pod \"7fd7656a-026c-4b86-a016-9e11446d27ae\" (UID: \"7fd7656a-026c-4b86-a016-9e11446d27ae\") " Oct 30 20:54:48.291138 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:48.291116 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd7656a-026c-4b86-a016-9e11446d27ae-maas-api-openshift-ca" pod "7fd7656a-026c-4b86-a016-9e11446d27ae" (UID: "7fd7656a-026c-4b86-a016-9e11446d27ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:48.292102 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:48.292076 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd7656a-026c-4b86-a016-9e11446d27ae-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:48.293319 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:48.293290 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd7656a-026c-4b86-a016-9e11446d27ae-kube-api-access-5dmfn" pod "7fd7656a-026c-4b86-a016-9e11446d27ae" (UID: "7fd7656a-026c-4b86-a016-9e11446d27ae"). InnerVolumeSpecName "kube-api-access-5dmfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:48.293467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:48.293344 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fd7656a-026c-4b86-a016-9e11446d27ae-maas-api-xks-ca" pod "7fd7656a-026c-4b86-a016-9e11446d27ae" (UID: "7fd7656a-026c-4b86-a016-9e11446d27ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:48.394153 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:48.394116 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd7656a-026c-4b86-a016-9e11446d27ae-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:48.394153 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:48.394144 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dmfn\" (UniqueName: \"kubernetes.io/projected/7fd7656a-026c-4b86-a016-9e11446d27ae-kube-api-access-5dmfn\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:48.727540 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:48.727507 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq"] Oct 30 20:54:48.946591 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:48.946561 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" Oct 30 20:54:48.946762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:48.946588 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw" event={"ID":"7fd7656a-026c-4b86-a016-9e11446d27ae","Type":"ContainerDied","Data":"a53c98c6718529109d2fc40763c362ef9b0728c3bd2f41cd83117c05cb22ae96"} Oct 30 20:54:48.946762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:54:48.946641 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a53c98c6718529109d2fc40763c362ef9b0728c3bd2f41cd83117c05cb22ae96" Oct 30 20:55:03.749168 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:03.749120 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" containerID="cri-o://acdb8fad8dac50a4b7fefff060442ff12cdbcbf9d2370d08b342716af9eadce1" gracePeriod=45 Oct 30 20:55:04.026962 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.026937 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:55:04.066156 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.066120 5755 generic.go:356] "Generic (PLEG): container finished" podID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerID="acdb8fad8dac50a4b7fefff060442ff12cdbcbf9d2370d08b342716af9eadce1" exitCode=0 Oct 30 20:55:04.066350 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.066175 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" Oct 30 20:55:04.066350 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.066206 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" event={"ID":"9614ceb3-8b56-4eb6-84cf-8904e13a1dda","Type":"ContainerDied","Data":"acdb8fad8dac50a4b7fefff060442ff12cdbcbf9d2370d08b342716af9eadce1"} Oct 30 20:55:04.066350 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.066264 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq" event={"ID":"9614ceb3-8b56-4eb6-84cf-8904e13a1dda","Type":"ContainerDied","Data":"d0b9adcce6f3376b9d0f8dcbbda9d3fcfb0b487110fd759beb5bf98977d019eb"} Oct 30 20:55:04.066350 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.066286 5755 scope.go:122] "RemoveContainer" containerID="acdb8fad8dac50a4b7fefff060442ff12cdbcbf9d2370d08b342716af9eadce1" Oct 30 20:55:04.076679 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.076661 5755 scope.go:122] "RemoveContainer" containerID="acdb8fad8dac50a4b7fefff060442ff12cdbcbf9d2370d08b342716af9eadce1" Oct 30 20:55:04.077000 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:55:04.076978 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acdb8fad8dac50a4b7fefff060442ff12cdbcbf9d2370d08b342716af9eadce1\": container with ID starting with acdb8fad8dac50a4b7fefff060442ff12cdbcbf9d2370d08b342716af9eadce1 not found: ID does not exist" containerID="acdb8fad8dac50a4b7fefff060442ff12cdbcbf9d2370d08b342716af9eadce1" Oct 30 20:55:04.077078 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.077013 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acdb8fad8dac50a4b7fefff060442ff12cdbcbf9d2370d08b342716af9eadce1"} err="failed to get container status \"acdb8fad8dac50a4b7fefff060442ff12cdbcbf9d2370d08b342716af9eadce1\": rpc error: code = NotFound desc = could not find container \"acdb8fad8dac50a4b7fefff060442ff12cdbcbf9d2370d08b342716af9eadce1\": container with ID starting with acdb8fad8dac50a4b7fefff060442ff12cdbcbf9d2370d08b342716af9eadce1 not found: ID does not exist" Oct 30 20:55:04.150633 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.150594 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-tls-certs\" (UniqueName: \"kubernetes.io/secret/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-tls-certs\") pod \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " Oct 30 20:55:04.150820 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.150704 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-model-cache\") pod \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " Oct 30 20:55:04.150820 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.150748 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-dshm\") pod \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " Oct 30 20:55:04.150820 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.150788 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-kube-api-access-dcg49\" (UniqueName: \"kubernetes.io/projected/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-kube-api-access-dcg49\") pod \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " Oct 30 20:55:04.151049 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.150825 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-tmp-dir\") pod \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " Oct 30 20:55:04.151049 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.150863 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-home\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-home\") pod \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\" (UID: \"9614ceb3-8b56-4eb6-84cf-8904e13a1dda\") " Oct 30 20:55:04.151049 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.150980 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-model-cache" pod "9614ceb3-8b56-4eb6-84cf-8904e13a1dda" (UID: "9614ceb3-8b56-4eb6-84cf-8904e13a1dda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:04.151253 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.151229 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-tmp-dir" pod "9614ceb3-8b56-4eb6-84cf-8904e13a1dda" (UID: "9614ceb3-8b56-4eb6-84cf-8904e13a1dda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:04.151366 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.151347 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-home" pod "9614ceb3-8b56-4eb6-84cf-8904e13a1dda" (UID: "9614ceb3-8b56-4eb6-84cf-8904e13a1dda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:04.151562 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.151536 5755 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-tmp-dir\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:04.151562 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.151556 5755 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-home\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:04.151707 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.151568 5755 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-model-cache\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:04.155666 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.154157 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-tls-certs" pod "9614ceb3-8b56-4eb6-84cf-8904e13a1dda" (UID: "9614ceb3-8b56-4eb6-84cf-8904e13a1dda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:04.155666 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.154284 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-kube-api-access-dcg49" pod "9614ceb3-8b56-4eb6-84cf-8904e13a1dda" (UID: "9614ceb3-8b56-4eb6-84cf-8904e13a1dda"). InnerVolumeSpecName "kube-api-access-dcg49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:04.158192 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.158164 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-dshm" pod "9614ceb3-8b56-4eb6-84cf-8904e13a1dda" (UID: "9614ceb3-8b56-4eb6-84cf-8904e13a1dda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:04.252808 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.252775 5755 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-tls-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:04.252808 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.252804 5755 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-dshm\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:04.252808 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.252817 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcg49\" (UniqueName: \"kubernetes.io/projected/9614ceb3-8b56-4eb6-84cf-8904e13a1dda-kube-api-access-dcg49\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:04.498287 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.498254 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq"] Oct 30 20:55:04.501477 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:04.501455 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-86d620/premium-w1-86d620-kserve-bf694fd9f-ccrrq"] Oct 30 20:55:05.216992 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:05.216954 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" path="/var/lib/kubelet/pods/9614ceb3-8b56-4eb6-84cf-8904e13a1dda/volumes" Oct 30 20:55:11.256620 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:55:11.256581 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:55:13.507907 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.507870 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc"] Oct 30 20:55:13.508799 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.508780 5755 cpu_manager.go:414] "containerMap: removing container" podUID="7fd7656a-026c-4b86-a016-9e11446d27ae" containerName="revoke-keys" Oct 30 20:55:13.508879 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.508800 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd7656a-026c-4b86-a016-9e11446d27ae" containerName="revoke-keys" Oct 30 20:55:13.508879 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.508842 5755 cpu_manager.go:414] "containerMap: removing container" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" Oct 30 20:55:13.508879 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.508850 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" Oct 30 20:55:13.508995 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.508984 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fd7656a-026c-4b86-a016-9e11446d27ae" containerName="revoke-keys" Oct 30 20:55:13.509037 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.509001 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="9614ceb3-8b56-4eb6-84cf-8904e13a1dda" containerName="main" Oct 30 20:55:13.512917 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.512892 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" Oct 30 20:55:13.561875 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.561839 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp"] Oct 30 20:55:13.566025 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.566002 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" Oct 30 20:55:13.594509 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.594476 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e6b94f-518c-49b1-a079-d589db93402a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ad6a291-s6prp\" (UID: \"32e6b94f-518c-49b1-a079-d589db93402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" Oct 30 20:55:13.594668 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.594516 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sxwf\" (UniqueName: \"kubernetes.io/projected/32e6b94f-518c-49b1-a079-d589db93402a-kube-api-access-9sxwf\") pod \"maas-api-revoke-sub-2ad6a291-s6prp\" (UID: \"32e6b94f-518c-49b1-a079-d589db93402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" Oct 30 20:55:13.594722 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.594700 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e6b94f-518c-49b1-a079-d589db93402a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ad6a291-s6prp\" (UID: \"32e6b94f-518c-49b1-a079-d589db93402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" Oct 30 20:55:13.594768 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.594740 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgdw8\" (UniqueName: \"kubernetes.io/projected/965dcdcd-dcea-4588-bec1-c4e8bd92785c-kube-api-access-kgdw8\") pod \"maas-api-revoke-sub-4f87cb69-lw9pc\" (UID: \"965dcdcd-dcea-4588-bec1-c4e8bd92785c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" Oct 30 20:55:13.594878 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.594862 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965dcdcd-dcea-4588-bec1-c4e8bd92785c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f87cb69-lw9pc\" (UID: \"965dcdcd-dcea-4588-bec1-c4e8bd92785c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" Oct 30 20:55:13.594939 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.594909 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965dcdcd-dcea-4588-bec1-c4e8bd92785c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f87cb69-lw9pc\" (UID: \"965dcdcd-dcea-4588-bec1-c4e8bd92785c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" Oct 30 20:55:13.607529 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.607499 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:55:13.613542 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.613172 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc"] Oct 30 20:55:13.615022 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.614995 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:55:13.622543 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.622267 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp"] Oct 30 20:55:13.696010 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.695973 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965dcdcd-dcea-4588-bec1-c4e8bd92785c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f87cb69-lw9pc\" (UID: \"965dcdcd-dcea-4588-bec1-c4e8bd92785c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" Oct 30 20:55:13.696188 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.696020 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965dcdcd-dcea-4588-bec1-c4e8bd92785c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f87cb69-lw9pc\" (UID: \"965dcdcd-dcea-4588-bec1-c4e8bd92785c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" Oct 30 20:55:13.696188 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.696182 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e6b94f-518c-49b1-a079-d589db93402a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ad6a291-s6prp\" (UID: \"32e6b94f-518c-49b1-a079-d589db93402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" Oct 30 20:55:13.696260 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.696210 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sxwf\" (UniqueName: \"kubernetes.io/projected/32e6b94f-518c-49b1-a079-d589db93402a-kube-api-access-9sxwf\") pod \"maas-api-revoke-sub-2ad6a291-s6prp\" (UID: \"32e6b94f-518c-49b1-a079-d589db93402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" Oct 30 20:55:13.696339 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.696309 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e6b94f-518c-49b1-a079-d589db93402a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ad6a291-s6prp\" (UID: \"32e6b94f-518c-49b1-a079-d589db93402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" Oct 30 20:55:13.696522 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.696355 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgdw8\" (UniqueName: \"kubernetes.io/projected/965dcdcd-dcea-4588-bec1-c4e8bd92785c-kube-api-access-kgdw8\") pod \"maas-api-revoke-sub-4f87cb69-lw9pc\" (UID: \"965dcdcd-dcea-4588-bec1-c4e8bd92785c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" Oct 30 20:55:13.696824 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.696800 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965dcdcd-dcea-4588-bec1-c4e8bd92785c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f87cb69-lw9pc\" (UID: \"965dcdcd-dcea-4588-bec1-c4e8bd92785c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" Oct 30 20:55:13.697024 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.697004 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e6b94f-518c-49b1-a079-d589db93402a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ad6a291-s6prp\" (UID: \"32e6b94f-518c-49b1-a079-d589db93402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" Oct 30 20:55:13.698725 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.698706 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e6b94f-518c-49b1-a079-d589db93402a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ad6a291-s6prp\" (UID: \"32e6b94f-518c-49b1-a079-d589db93402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" Oct 30 20:55:13.698784 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.698742 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965dcdcd-dcea-4588-bec1-c4e8bd92785c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f87cb69-lw9pc\" (UID: \"965dcdcd-dcea-4588-bec1-c4e8bd92785c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" Oct 30 20:55:13.755003 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.754974 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sxwf\" (UniqueName: \"kubernetes.io/projected/32e6b94f-518c-49b1-a079-d589db93402a-kube-api-access-9sxwf\") pod \"maas-api-revoke-sub-2ad6a291-s6prp\" (UID: \"32e6b94f-518c-49b1-a079-d589db93402a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" Oct 30 20:55:13.755240 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.755222 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgdw8\" (UniqueName: \"kubernetes.io/projected/965dcdcd-dcea-4588-bec1-c4e8bd92785c-kube-api-access-kgdw8\") pod \"maas-api-revoke-sub-4f87cb69-lw9pc\" (UID: \"965dcdcd-dcea-4588-bec1-c4e8bd92785c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" Oct 30 20:55:13.828146 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.828061 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" Oct 30 20:55:13.879657 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:13.879620 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" Oct 30 20:55:14.208543 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:14.208515 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc"] Oct 30 20:55:14.209100 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:55:14.209077 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965dcdcd_dcea_4588_bec1_c4e8bd92785c.slice/crio-3ea5e653eca494997893f008d0196a3b9464a1522a9482955a5d795bb342f617 WatchSource:0}: Error finding container 3ea5e653eca494997893f008d0196a3b9464a1522a9482955a5d795bb342f617: Status 404 returned error can't find the container with id 3ea5e653eca494997893f008d0196a3b9464a1522a9482955a5d795bb342f617 Oct 30 20:55:14.270863 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:14.270838 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp"] Oct 30 20:55:14.271672 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:55:14.271644 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e6b94f_518c_49b1_a079_d589db93402a.slice/crio-dbd3c87b33b341bd9f070bddde3dfead3050342edc9836d53194e9a02ffea7b0 WatchSource:0}: Error finding container dbd3c87b33b341bd9f070bddde3dfead3050342edc9836d53194e9a02ffea7b0: Status 404 returned error can't find the container with id dbd3c87b33b341bd9f070bddde3dfead3050342edc9836d53194e9a02ffea7b0 Oct 30 20:55:15.144993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:15.144958 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" event={"ID":"965dcdcd-dcea-4588-bec1-c4e8bd92785c","Type":"ContainerStarted","Data":"dcadbc61171cb2c64151b5a8af952fdc783b5795cc3f21d7bdc520a6b3f3f8a0"} Oct 30 20:55:15.144993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:15.145022 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" event={"ID":"965dcdcd-dcea-4588-bec1-c4e8bd92785c","Type":"ContainerStarted","Data":"3ea5e653eca494997893f008d0196a3b9464a1522a9482955a5d795bb342f617"} Oct 30 20:55:15.146971 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:15.146938 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" event={"ID":"32e6b94f-518c-49b1-a079-d589db93402a","Type":"ContainerStarted","Data":"d3611f6c63d4df8442a2616d16f4d9868dc036d2e42a9ec08e2cfcce190916c8"} Oct 30 20:55:15.147107 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:15.146993 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" event={"ID":"32e6b94f-518c-49b1-a079-d589db93402a","Type":"ContainerStarted","Data":"dbd3c87b33b341bd9f070bddde3dfead3050342edc9836d53194e9a02ffea7b0"} Oct 30 20:55:15.261234 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:15.261186 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" podStartSLOduration=2.261168499 podStartE2EDuration="2.261168499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:15.259367191 +0000 UTC m=+1678.185924106" watchObservedRunningTime="2026-10-30 20:55:15.261168499 +0000 UTC m=+1678.187725414" Oct 30 20:55:15.378643 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:15.378594 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" podStartSLOduration=2.37858094 podStartE2EDuration="2.37858094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:15.376934256 +0000 UTC m=+1678.303491197" watchObservedRunningTime="2026-10-30 20:55:15.37858094 +0000 UTC m=+1678.305137854" Oct 30 20:55:16.155888 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:16.155811 5755 generic.go:356] "Generic (PLEG): container finished" podID="32e6b94f-518c-49b1-a079-d589db93402a" containerID="d3611f6c63d4df8442a2616d16f4d9868dc036d2e42a9ec08e2cfcce190916c8" exitCode=0 Oct 30 20:55:16.156370 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:16.155893 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" event={"ID":"32e6b94f-518c-49b1-a079-d589db93402a","Type":"ContainerDied","Data":"d3611f6c63d4df8442a2616d16f4d9868dc036d2e42a9ec08e2cfcce190916c8"} Oct 30 20:55:16.157638 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:16.157613 5755 generic.go:356] "Generic (PLEG): container finished" podID="965dcdcd-dcea-4588-bec1-c4e8bd92785c" containerID="dcadbc61171cb2c64151b5a8af952fdc783b5795cc3f21d7bdc520a6b3f3f8a0" exitCode=0 Oct 30 20:55:16.157757 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:16.157688 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" event={"ID":"965dcdcd-dcea-4588-bec1-c4e8bd92785c","Type":"ContainerDied","Data":"dcadbc61171cb2c64151b5a8af952fdc783b5795cc3f21d7bdc520a6b3f3f8a0"} Oct 30 20:55:17.334052 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.334023 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" Oct 30 20:55:17.376972 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.376948 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" Oct 30 20:55:17.466420 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.466315 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32e6b94f-518c-49b1-a079-d589db93402a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e6b94f-518c-49b1-a079-d589db93402a-maas-api-xks-ca\") pod \"32e6b94f-518c-49b1-a079-d589db93402a\" (UID: \"32e6b94f-518c-49b1-a079-d589db93402a\") " Oct 30 20:55:17.466580 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.466438 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965dcdcd-dcea-4588-bec1-c4e8bd92785c-kube-api-access-kgdw8\" (UniqueName: \"kubernetes.io/projected/965dcdcd-dcea-4588-bec1-c4e8bd92785c-kube-api-access-kgdw8\") pod \"965dcdcd-dcea-4588-bec1-c4e8bd92785c\" (UID: \"965dcdcd-dcea-4588-bec1-c4e8bd92785c\") " Oct 30 20:55:17.466666 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.466626 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/965dcdcd-dcea-4588-bec1-c4e8bd92785c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965dcdcd-dcea-4588-bec1-c4e8bd92785c-maas-api-openshift-ca\") pod \"965dcdcd-dcea-4588-bec1-c4e8bd92785c\" (UID: \"965dcdcd-dcea-4588-bec1-c4e8bd92785c\") " Oct 30 20:55:17.467006 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.466978 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965dcdcd-dcea-4588-bec1-c4e8bd92785c-maas-api-openshift-ca" pod "965dcdcd-dcea-4588-bec1-c4e8bd92785c" (UID: "965dcdcd-dcea-4588-bec1-c4e8bd92785c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:17.468036 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.467671 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e6b94f-518c-49b1-a079-d589db93402a-kube-api-access-9sxwf\" (UniqueName: \"kubernetes.io/projected/32e6b94f-518c-49b1-a079-d589db93402a-kube-api-access-9sxwf\") pod \"32e6b94f-518c-49b1-a079-d589db93402a\" (UID: \"32e6b94f-518c-49b1-a079-d589db93402a\") " Oct 30 20:55:17.468036 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.467757 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/965dcdcd-dcea-4588-bec1-c4e8bd92785c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965dcdcd-dcea-4588-bec1-c4e8bd92785c-maas-api-xks-ca\") pod \"965dcdcd-dcea-4588-bec1-c4e8bd92785c\" (UID: \"965dcdcd-dcea-4588-bec1-c4e8bd92785c\") " Oct 30 20:55:17.468208 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.468158 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32e6b94f-518c-49b1-a079-d589db93402a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e6b94f-518c-49b1-a079-d589db93402a-maas-api-openshift-ca\") pod \"32e6b94f-518c-49b1-a079-d589db93402a\" (UID: \"32e6b94f-518c-49b1-a079-d589db93402a\") " Oct 30 20:55:17.469311 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.468907 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965dcdcd-dcea-4588-bec1-c4e8bd92785c-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:17.469311 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.469230 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e6b94f-518c-49b1-a079-d589db93402a-maas-api-xks-ca" pod "32e6b94f-518c-49b1-a079-d589db93402a" (UID: "32e6b94f-518c-49b1-a079-d589db93402a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:17.469311 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.469262 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965dcdcd-dcea-4588-bec1-c4e8bd92785c-kube-api-access-kgdw8" pod "965dcdcd-dcea-4588-bec1-c4e8bd92785c" (UID: "965dcdcd-dcea-4588-bec1-c4e8bd92785c"). InnerVolumeSpecName "kube-api-access-kgdw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:17.469311 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.469290 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e6b94f-518c-49b1-a079-d589db93402a-maas-api-openshift-ca" pod "32e6b94f-518c-49b1-a079-d589db93402a" (UID: "32e6b94f-518c-49b1-a079-d589db93402a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:17.471552 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.471525 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/965dcdcd-dcea-4588-bec1-c4e8bd92785c-maas-api-xks-ca" pod "965dcdcd-dcea-4588-bec1-c4e8bd92785c" (UID: "965dcdcd-dcea-4588-bec1-c4e8bd92785c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:17.472628 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.472591 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e6b94f-518c-49b1-a079-d589db93402a-kube-api-access-9sxwf" pod "32e6b94f-518c-49b1-a079-d589db93402a" (UID: "32e6b94f-518c-49b1-a079-d589db93402a"). InnerVolumeSpecName "kube-api-access-9sxwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:17.572349 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.572317 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sxwf\" (UniqueName: \"kubernetes.io/projected/32e6b94f-518c-49b1-a079-d589db93402a-kube-api-access-9sxwf\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:17.572349 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.572348 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965dcdcd-dcea-4588-bec1-c4e8bd92785c-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:17.572569 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.572358 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e6b94f-518c-49b1-a079-d589db93402a-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:17.572569 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.572367 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e6b94f-518c-49b1-a079-d589db93402a-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:17.572569 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:17.572376 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgdw8\" (UniqueName: \"kubernetes.io/projected/965dcdcd-dcea-4588-bec1-c4e8bd92785c-kube-api-access-kgdw8\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:18.172132 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:18.172099 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" Oct 30 20:55:18.172132 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:18.172105 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc" event={"ID":"965dcdcd-dcea-4588-bec1-c4e8bd92785c","Type":"ContainerDied","Data":"3ea5e653eca494997893f008d0196a3b9464a1522a9482955a5d795bb342f617"} Oct 30 20:55:18.172381 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:18.172163 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ea5e653eca494997893f008d0196a3b9464a1522a9482955a5d795bb342f617" Oct 30 20:55:18.173753 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:18.173729 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" Oct 30 20:55:18.173871 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:18.173724 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp" event={"ID":"32e6b94f-518c-49b1-a079-d589db93402a","Type":"ContainerDied","Data":"dbd3c87b33b341bd9f070bddde3dfead3050342edc9836d53194e9a02ffea7b0"} Oct 30 20:55:18.173871 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:18.173857 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbd3c87b33b341bd9f070bddde3dfead3050342edc9836d53194e9a02ffea7b0" Oct 30 20:55:26.381044 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:26.381004 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9"] Oct 30 20:55:26.428066 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:26.428037 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b32941-69nb9"] Oct 30 20:55:27.212812 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:27.212779 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1984d4d-e168-4633-96e2-7bca3856afc5" path="/var/lib/kubelet/pods/e1984d4d-e168-4633-96e2-7bca3856afc5/volumes" Oct 30 20:55:36.493929 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:36.493894 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj"] Oct 30 20:55:36.494423 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:36.494151 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" podUID="9847155d-936e-482a-a172-d79a44533b2c" containerName="istio-proxy" containerID="cri-o://592c2221434b29acdac0e4d52dac188eb1d0f6a31bdfa7dcd8ed17ff88c81722" gracePeriod=30 Oct 30 20:55:36.518659 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:36.518628 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf"] Oct 30 20:55:36.518885 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:36.518847 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" podUID="3d477ecb-658f-4ce9-af37-3ca9f557b889" containerName="istio-proxy" containerID="cri-o://a9ce23bdfddc5d30b1ac256e5fea02f7df0bc45065fc2e561220e30ab209efb4" gracePeriod=30 Oct 30 20:55:38.487974 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:38.487934 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:55:38.488361 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:38.487993 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" podUID="9847155d-936e-482a-a172-d79a44533b2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:55:39.420549 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:39.420513 5755 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:55:39.420711 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:39.420575 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" podUID="3d477ecb-658f-4ce9-af37-3ca9f557b889" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:55:41.256375 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:55:41.256339 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:55:41.808946 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.808916 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:55:41.828611 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.828585 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:55:41.904129 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904095 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d477ecb-658f-4ce9-af37-3ca9f557b889-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d477ecb-658f-4ce9-af37-3ca9f557b889-istiod-ca-cert\") pod \"3d477ecb-658f-4ce9-af37-3ca9f557b889\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " Oct 30 20:55:41.904332 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904136 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-istio-data\") pod \"9847155d-936e-482a-a172-d79a44533b2c\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " Oct 30 20:55:41.904332 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904179 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-workload-socket\") pod \"9847155d-936e-482a-a172-d79a44533b2c\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " Oct 30 20:55:41.904332 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904237 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-workload-certs\") pod \"9847155d-936e-482a-a172-d79a44533b2c\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " Oct 30 20:55:41.904332 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904273 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-data\") pod \"3d477ecb-658f-4ce9-af37-3ca9f557b889\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " Oct 30 20:55:41.904332 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904297 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-envoy\") pod \"3d477ecb-658f-4ce9-af37-3ca9f557b889\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " Oct 30 20:55:41.904723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904349 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9847155d-936e-482a-a172-d79a44533b2c-istio-token\" (UniqueName: \"kubernetes.io/projected/9847155d-936e-482a-a172-d79a44533b2c-istio-token\") pod \"9847155d-936e-482a-a172-d79a44533b2c\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " Oct 30 20:55:41.904723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904447 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9847155d-936e-482a-a172-d79a44533b2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9847155d-936e-482a-a172-d79a44533b2c-istiod-ca-cert\") pod \"9847155d-936e-482a-a172-d79a44533b2c\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " Oct 30 20:55:41.904723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904480 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-istio-envoy\") pod \"9847155d-936e-482a-a172-d79a44533b2c\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " Oct 30 20:55:41.904723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904507 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-podinfo\") pod \"3d477ecb-658f-4ce9-af37-3ca9f557b889\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " Oct 30 20:55:41.904723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904531 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-workload-certs" pod "9847155d-936e-482a-a172-d79a44533b2c" (UID: "9847155d-936e-482a-a172-d79a44533b2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:41.904723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904572 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-credential-socket\") pod \"9847155d-936e-482a-a172-d79a44533b2c\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " Oct 30 20:55:41.904723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904541 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-istio-data" pod "9847155d-936e-482a-a172-d79a44533b2c" (UID: "9847155d-936e-482a-a172-d79a44533b2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:41.904723 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904608 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-workload-socket" pod "9847155d-936e-482a-a172-d79a44533b2c" (UID: "9847155d-936e-482a-a172-d79a44533b2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:41.905082 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904856 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-data" pod "3d477ecb-658f-4ce9-af37-3ca9f557b889" (UID: "3d477ecb-658f-4ce9-af37-3ca9f557b889"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:41.905082 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904976 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9847155d-936e-482a-a172-d79a44533b2c-istiod-ca-cert" pod "9847155d-936e-482a-a172-d79a44533b2c" (UID: "9847155d-936e-482a-a172-d79a44533b2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:41.905082 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904986 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d477ecb-658f-4ce9-af37-3ca9f557b889-istiod-ca-cert" pod "3d477ecb-658f-4ce9-af37-3ca9f557b889" (UID: "3d477ecb-658f-4ce9-af37-3ca9f557b889"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:41.905082 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904989 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-workload-socket\") pod \"3d477ecb-658f-4ce9-af37-3ca9f557b889\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " Oct 30 20:55:41.905082 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.904997 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-credential-socket" pod "9847155d-936e-482a-a172-d79a44533b2c" (UID: "9847155d-936e-482a-a172-d79a44533b2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:41.905082 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.905064 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-workload-certs\") pod \"3d477ecb-658f-4ce9-af37-3ca9f557b889\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " Oct 30 20:55:41.905575 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.905151 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-credential-socket\") pod \"3d477ecb-658f-4ce9-af37-3ca9f557b889\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " Oct 30 20:55:41.905575 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.905159 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-workload-socket" pod "3d477ecb-658f-4ce9-af37-3ca9f557b889" (UID: "3d477ecb-658f-4ce9-af37-3ca9f557b889"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:41.905575 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.905212 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9847155d-936e-482a-a172-d79a44533b2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9847155d-936e-482a-a172-d79a44533b2c-istio-podinfo\") pod \"9847155d-936e-482a-a172-d79a44533b2c\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " Oct 30 20:55:41.905575 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.905253 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-token\" (UniqueName: \"kubernetes.io/projected/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-token\") pod \"3d477ecb-658f-4ce9-af37-3ca9f557b889\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " Oct 30 20:55:41.905575 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.905324 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9847155d-936e-482a-a172-d79a44533b2c-kube-api-access-9bs4d\" (UniqueName: \"kubernetes.io/projected/9847155d-936e-482a-a172-d79a44533b2c-kube-api-access-9bs4d\") pod \"9847155d-936e-482a-a172-d79a44533b2c\" (UID: \"9847155d-936e-482a-a172-d79a44533b2c\") " Oct 30 20:55:41.905575 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.905500 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-workload-certs" pod "3d477ecb-658f-4ce9-af37-3ca9f557b889" (UID: "3d477ecb-658f-4ce9-af37-3ca9f557b889"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:41.905891 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.905603 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-credential-socket" pod "3d477ecb-658f-4ce9-af37-3ca9f557b889" (UID: "3d477ecb-658f-4ce9-af37-3ca9f557b889"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:41.905891 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.905674 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d477ecb-658f-4ce9-af37-3ca9f557b889-kube-api-access-xg2f5\" (UniqueName: \"kubernetes.io/projected/3d477ecb-658f-4ce9-af37-3ca9f557b889-kube-api-access-xg2f5\") pod \"3d477ecb-658f-4ce9-af37-3ca9f557b889\" (UID: \"3d477ecb-658f-4ce9-af37-3ca9f557b889\") " Oct 30 20:55:41.906417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.906372 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:41.906581 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.906418 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9847155d-936e-482a-a172-d79a44533b2c-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:41.906581 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.906435 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:41.906581 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.906449 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:41.906581 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.906464 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:41.906581 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.906478 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:41.906581 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.906493 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d477ecb-658f-4ce9-af37-3ca9f557b889-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:41.906581 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.906508 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:41.906581 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.906525 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:41.906581 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.906538 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:41.907430 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.907374 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-istio-envoy" pod "9847155d-936e-482a-a172-d79a44533b2c" (UID: "9847155d-936e-482a-a172-d79a44533b2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:41.907634 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.907603 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-podinfo" pod "3d477ecb-658f-4ce9-af37-3ca9f557b889" (UID: "3d477ecb-658f-4ce9-af37-3ca9f557b889"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:55:41.907739 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.907686 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-envoy" pod "3d477ecb-658f-4ce9-af37-3ca9f557b889" (UID: "3d477ecb-658f-4ce9-af37-3ca9f557b889"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:41.907790 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.907737 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9847155d-936e-482a-a172-d79a44533b2c-istio-token" pod "9847155d-936e-482a-a172-d79a44533b2c" (UID: "9847155d-936e-482a-a172-d79a44533b2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:41.907998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.907976 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-token" pod "3d477ecb-658f-4ce9-af37-3ca9f557b889" (UID: "3d477ecb-658f-4ce9-af37-3ca9f557b889"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:41.908619 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.908600 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d477ecb-658f-4ce9-af37-3ca9f557b889-kube-api-access-xg2f5" pod "3d477ecb-658f-4ce9-af37-3ca9f557b889" (UID: "3d477ecb-658f-4ce9-af37-3ca9f557b889"). InnerVolumeSpecName "kube-api-access-xg2f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:41.908992 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.908979 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9847155d-936e-482a-a172-d79a44533b2c-istio-podinfo" pod "9847155d-936e-482a-a172-d79a44533b2c" (UID: "9847155d-936e-482a-a172-d79a44533b2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:55:41.909062 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:41.909045 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9847155d-936e-482a-a172-d79a44533b2c-kube-api-access-9bs4d" pod "9847155d-936e-482a-a172-d79a44533b2c" (UID: "9847155d-936e-482a-a172-d79a44533b2c"). InnerVolumeSpecName "kube-api-access-9bs4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:42.013444 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.008663 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:42.013444 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.008701 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9847155d-936e-482a-a172-d79a44533b2c-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:42.013444 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.008715 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9847155d-936e-482a-a172-d79a44533b2c-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:42.013444 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.008731 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:42.013444 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.008747 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9847155d-936e-482a-a172-d79a44533b2c-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:42.013444 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.008761 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d477ecb-658f-4ce9-af37-3ca9f557b889-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:42.013444 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.008776 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bs4d\" (UniqueName: \"kubernetes.io/projected/9847155d-936e-482a-a172-d79a44533b2c-kube-api-access-9bs4d\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:42.013444 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.008792 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg2f5\" (UniqueName: \"kubernetes.io/projected/3d477ecb-658f-4ce9-af37-3ca9f557b889-kube-api-access-xg2f5\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:42.368478 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.368446 5755 generic.go:356] "Generic (PLEG): container finished" podID="9847155d-936e-482a-a172-d79a44533b2c" containerID="592c2221434b29acdac0e4d52dac188eb1d0f6a31bdfa7dcd8ed17ff88c81722" exitCode=0 Oct 30 20:55:42.369083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.368494 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" Oct 30 20:55:42.369083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.368534 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" event={"ID":"9847155d-936e-482a-a172-d79a44533b2c","Type":"ContainerDied","Data":"592c2221434b29acdac0e4d52dac188eb1d0f6a31bdfa7dcd8ed17ff88c81722"} Oct 30 20:55:42.369083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.368596 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj" event={"ID":"9847155d-936e-482a-a172-d79a44533b2c","Type":"ContainerDied","Data":"a3fdf5abdc696ef5dcf6c2eca0c35c50d4d1e1c1156ec4f59949ae38cd7cf731"} Oct 30 20:55:42.369083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.368631 5755 scope.go:122] "RemoveContainer" containerID="592c2221434b29acdac0e4d52dac188eb1d0f6a31bdfa7dcd8ed17ff88c81722" Oct 30 20:55:42.370403 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.370366 5755 generic.go:356] "Generic (PLEG): container finished" podID="3d477ecb-658f-4ce9-af37-3ca9f557b889" containerID="a9ce23bdfddc5d30b1ac256e5fea02f7df0bc45065fc2e561220e30ab209efb4" exitCode=0 Oct 30 20:55:42.370547 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.370445 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" Oct 30 20:55:42.370547 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.370470 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" event={"ID":"3d477ecb-658f-4ce9-af37-3ca9f557b889","Type":"ContainerDied","Data":"a9ce23bdfddc5d30b1ac256e5fea02f7df0bc45065fc2e561220e30ab209efb4"} Oct 30 20:55:42.370547 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.370501 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf" event={"ID":"3d477ecb-658f-4ce9-af37-3ca9f557b889","Type":"ContainerDied","Data":"7d4c50f693c044a498193a003662fe258e835f15f21914c381ac6b9672572b30"} Oct 30 20:55:42.379894 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.379877 5755 scope.go:122] "RemoveContainer" containerID="592c2221434b29acdac0e4d52dac188eb1d0f6a31bdfa7dcd8ed17ff88c81722" Oct 30 20:55:42.380255 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:55:42.380203 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592c2221434b29acdac0e4d52dac188eb1d0f6a31bdfa7dcd8ed17ff88c81722\": container with ID starting with 592c2221434b29acdac0e4d52dac188eb1d0f6a31bdfa7dcd8ed17ff88c81722 not found: ID does not exist" containerID="592c2221434b29acdac0e4d52dac188eb1d0f6a31bdfa7dcd8ed17ff88c81722" Oct 30 20:55:42.380324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.380239 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592c2221434b29acdac0e4d52dac188eb1d0f6a31bdfa7dcd8ed17ff88c81722"} err="failed to get container status \"592c2221434b29acdac0e4d52dac188eb1d0f6a31bdfa7dcd8ed17ff88c81722\": rpc error: code = NotFound desc = could not find container \"592c2221434b29acdac0e4d52dac188eb1d0f6a31bdfa7dcd8ed17ff88c81722\": container with ID starting with 592c2221434b29acdac0e4d52dac188eb1d0f6a31bdfa7dcd8ed17ff88c81722 not found: ID does not exist" Oct 30 20:55:42.380324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.380273 5755 scope.go:122] "RemoveContainer" containerID="a9ce23bdfddc5d30b1ac256e5fea02f7df0bc45065fc2e561220e30ab209efb4" Oct 30 20:55:42.390475 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.390456 5755 scope.go:122] "RemoveContainer" containerID="a9ce23bdfddc5d30b1ac256e5fea02f7df0bc45065fc2e561220e30ab209efb4" Oct 30 20:55:42.390732 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:55:42.390711 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ce23bdfddc5d30b1ac256e5fea02f7df0bc45065fc2e561220e30ab209efb4\": container with ID starting with a9ce23bdfddc5d30b1ac256e5fea02f7df0bc45065fc2e561220e30ab209efb4 not found: ID does not exist" containerID="a9ce23bdfddc5d30b1ac256e5fea02f7df0bc45065fc2e561220e30ab209efb4" Oct 30 20:55:42.390771 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.390740 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ce23bdfddc5d30b1ac256e5fea02f7df0bc45065fc2e561220e30ab209efb4"} err="failed to get container status \"a9ce23bdfddc5d30b1ac256e5fea02f7df0bc45065fc2e561220e30ab209efb4\": rpc error: code = NotFound desc = could not find container \"a9ce23bdfddc5d30b1ac256e5fea02f7df0bc45065fc2e561220e30ab209efb4\": container with ID starting with a9ce23bdfddc5d30b1ac256e5fea02f7df0bc45065fc2e561220e30ab209efb4 not found: ID does not exist" Oct 30 20:55:42.540602 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.540572 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf"] Oct 30 20:55:42.588149 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.588111 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-tp2qf"] Oct 30 20:55:42.657681 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.657605 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj"] Oct 30 20:55:42.661298 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:42.661273 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-wkmlj"] Oct 30 20:55:43.222002 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:43.221962 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d477ecb-658f-4ce9-af37-3ca9f557b889" path="/var/lib/kubelet/pods/3d477ecb-658f-4ce9-af37-3ca9f557b889/volumes" Oct 30 20:55:43.223216 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:55:43.223191 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9847155d-936e-482a-a172-d79a44533b2c" path="/var/lib/kubelet/pods/9847155d-936e-482a-a172-d79a44533b2c/volumes" Oct 30 20:56:02.880901 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.880863 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv"] Oct 30 20:56:02.882168 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.882142 5755 cpu_manager.go:414] "containerMap: removing container" podUID="32e6b94f-518c-49b1-a079-d589db93402a" containerName="revoke-keys" Oct 30 20:56:02.882306 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.882294 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e6b94f-518c-49b1-a079-d589db93402a" containerName="revoke-keys" Oct 30 20:56:02.882422 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.882411 5755 cpu_manager.go:414] "containerMap: removing container" podUID="3d477ecb-658f-4ce9-af37-3ca9f557b889" containerName="istio-proxy" Oct 30 20:56:02.882502 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.882492 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d477ecb-658f-4ce9-af37-3ca9f557b889" containerName="istio-proxy" Oct 30 20:56:02.882620 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.882611 5755 cpu_manager.go:414] "containerMap: removing container" podUID="965dcdcd-dcea-4588-bec1-c4e8bd92785c" containerName="revoke-keys" Oct 30 20:56:02.882692 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.882683 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="965dcdcd-dcea-4588-bec1-c4e8bd92785c" containerName="revoke-keys" Oct 30 20:56:02.882763 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.882755 5755 cpu_manager.go:414] "containerMap: removing container" podUID="9847155d-936e-482a-a172-d79a44533b2c" containerName="istio-proxy" Oct 30 20:56:02.882824 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.882815 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9847155d-936e-482a-a172-d79a44533b2c" containerName="istio-proxy" Oct 30 20:56:02.883035 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.883025 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="965dcdcd-dcea-4588-bec1-c4e8bd92785c" containerName="revoke-keys" Oct 30 20:56:02.883109 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.883100 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="9847155d-936e-482a-a172-d79a44533b2c" containerName="istio-proxy" Oct 30 20:56:02.883176 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.883168 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e6b94f-518c-49b1-a079-d589db93402a" containerName="revoke-keys" Oct 30 20:56:02.883250 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.883241 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d477ecb-658f-4ce9-af37-3ca9f557b889" containerName="istio-proxy" Oct 30 20:56:02.890659 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.890637 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv" Oct 30 20:56:02.944205 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.944175 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv"] Oct 30 20:56:02.989209 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.989179 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:56:02.989360 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:02.989331 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:56:03.027258 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:03.027218 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5433df-a320-41cd-962f-240640c4af39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv\" (UID: \"7a5433df-a320-41cd-962f-240640c4af39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv" Oct 30 20:56:03.027258 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:03.027261 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sm57\" (UniqueName: \"kubernetes.io/projected/7a5433df-a320-41cd-962f-240640c4af39-kube-api-access-8sm57\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv\" (UID: \"7a5433df-a320-41cd-962f-240640c4af39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv" Oct 30 20:56:03.027507 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:03.027443 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5433df-a320-41cd-962f-240640c4af39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv\" (UID: \"7a5433df-a320-41cd-962f-240640c4af39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv" Oct 30 20:56:03.129338 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:03.129305 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5433df-a320-41cd-962f-240640c4af39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv\" (UID: \"7a5433df-a320-41cd-962f-240640c4af39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv" Oct 30 20:56:03.129584 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:03.129565 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5433df-a320-41cd-962f-240640c4af39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv\" (UID: \"7a5433df-a320-41cd-962f-240640c4af39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv" Oct 30 20:56:03.129636 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:03.129597 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sm57\" (UniqueName: \"kubernetes.io/projected/7a5433df-a320-41cd-962f-240640c4af39-kube-api-access-8sm57\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv\" (UID: \"7a5433df-a320-41cd-962f-240640c4af39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv" Oct 30 20:56:03.130164 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:03.130136 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5433df-a320-41cd-962f-240640c4af39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv\" (UID: \"7a5433df-a320-41cd-962f-240640c4af39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv" Oct 30 20:56:03.131981 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:03.131934 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5433df-a320-41cd-962f-240640c4af39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv\" (UID: \"7a5433df-a320-41cd-962f-240640c4af39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv" Oct 30 20:56:03.187813 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:03.187771 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sm57\" (UniqueName: \"kubernetes.io/projected/7a5433df-a320-41cd-962f-240640c4af39-kube-api-access-8sm57\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv\" (UID: \"7a5433df-a320-41cd-962f-240640c4af39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv" Oct 30 20:56:03.212811 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:03.212784 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv" Oct 30 20:56:03.593212 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:56:03.593183 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5433df_a320_41cd_962f_240640c4af39.slice/crio-b2012bbc412cad552a6b569ea871e159a51a227a4e454e1ee69763f229118cd5 WatchSource:0}: Error finding container b2012bbc412cad552a6b569ea871e159a51a227a4e454e1ee69763f229118cd5: Status 404 returned error can't find the container with id b2012bbc412cad552a6b569ea871e159a51a227a4e454e1ee69763f229118cd5 Oct 30 20:56:03.593920 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:03.593897 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv"] Oct 30 20:56:04.585092 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:04.585060 5755 generic.go:356] "Generic (PLEG): container finished" podID="7a5433df-a320-41cd-962f-240640c4af39" containerID="0aee27e23f26f33fb8b8fa01eea5f6775ea14ecfeacc9ed45c360966da15f165" exitCode=0 Oct 30 20:56:04.585555 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:04.585143 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv" event={"ID":"7a5433df-a320-41cd-962f-240640c4af39","Type":"ContainerDied","Data":"0aee27e23f26f33fb8b8fa01eea5f6775ea14ecfeacc9ed45c360966da15f165"} Oct 30 20:56:04.585555 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:04.585193 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv" event={"ID":"7a5433df-a320-41cd-962f-240640c4af39","Type":"ContainerStarted","Data":"b2012bbc412cad552a6b569ea871e159a51a227a4e454e1ee69763f229118cd5"} Oct 30 20:56:05.763245 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:05.763221 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv" Oct 30 20:56:05.869810 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:05.869721 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a5433df-a320-41cd-962f-240640c4af39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5433df-a320-41cd-962f-240640c4af39-maas-api-openshift-ca\") pod \"7a5433df-a320-41cd-962f-240640c4af39\" (UID: \"7a5433df-a320-41cd-962f-240640c4af39\") " Oct 30 20:56:05.869977 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:05.869842 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a5433df-a320-41cd-962f-240640c4af39-kube-api-access-8sm57\" (UniqueName: \"kubernetes.io/projected/7a5433df-a320-41cd-962f-240640c4af39-kube-api-access-8sm57\") pod \"7a5433df-a320-41cd-962f-240640c4af39\" (UID: \"7a5433df-a320-41cd-962f-240640c4af39\") " Oct 30 20:56:05.869977 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:05.869917 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a5433df-a320-41cd-962f-240640c4af39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5433df-a320-41cd-962f-240640c4af39-maas-api-xks-ca\") pod \"7a5433df-a320-41cd-962f-240640c4af39\" (UID: \"7a5433df-a320-41cd-962f-240640c4af39\") " Oct 30 20:56:05.870170 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:05.870141 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a5433df-a320-41cd-962f-240640c4af39-maas-api-openshift-ca" pod "7a5433df-a320-41cd-962f-240640c4af39" (UID: "7a5433df-a320-41cd-962f-240640c4af39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:05.870556 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:05.870504 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5433df-a320-41cd-962f-240640c4af39-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:05.872101 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:05.872077 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a5433df-a320-41cd-962f-240640c4af39-maas-api-xks-ca" pod "7a5433df-a320-41cd-962f-240640c4af39" (UID: "7a5433df-a320-41cd-962f-240640c4af39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:56:05.872230 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:05.872193 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5433df-a320-41cd-962f-240640c4af39-kube-api-access-8sm57" pod "7a5433df-a320-41cd-962f-240640c4af39" (UID: "7a5433df-a320-41cd-962f-240640c4af39"). InnerVolumeSpecName "kube-api-access-8sm57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:05.972353 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:05.972317 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5433df-a320-41cd-962f-240640c4af39-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:05.972353 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:05.972351 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sm57\" (UniqueName: \"kubernetes.io/projected/7a5433df-a320-41cd-962f-240640c4af39-kube-api-access-8sm57\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:06.601403 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:06.601359 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv" Oct 30 20:56:06.601575 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:06.601415 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv" event={"ID":"7a5433df-a320-41cd-962f-240640c4af39","Type":"ContainerDied","Data":"b2012bbc412cad552a6b569ea871e159a51a227a4e454e1ee69763f229118cd5"} Oct 30 20:56:06.601575 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:06.601515 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2012bbc412cad552a6b569ea871e159a51a227a4e454e1ee69763f229118cd5" Oct 30 20:56:11.256151 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:56:11.256111 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:56:28.867758 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:28.867718 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42"] Oct 30 20:56:28.868331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:28.868246 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" podUID="85e81941-ca08-4234-9288-0dd4ecddf92f" containerName="maas-api" containerID="cri-o://b645c249272cb0b0ee9b701cae19ccdace31539dec6c266f8192d3786391d2cc" gracePeriod=30 Oct 30 20:56:29.275703 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.275679 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:56:29.455013 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.454872 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e81941-ca08-4234-9288-0dd4ecddf92f-kube-api-access-l28bw\" (UniqueName: \"kubernetes.io/projected/85e81941-ca08-4234-9288-0dd4ecddf92f-kube-api-access-l28bw\") pod \"85e81941-ca08-4234-9288-0dd4ecddf92f\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " Oct 30 20:56:29.455013 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.454965 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85e81941-ca08-4234-9288-0dd4ecddf92f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e81941-ca08-4234-9288-0dd4ecddf92f-openshift-service-ca\") pod \"85e81941-ca08-4234-9288-0dd4ecddf92f\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " Oct 30 20:56:29.455211 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.455016 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-metrics-certs\") pod \"85e81941-ca08-4234-9288-0dd4ecddf92f\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " Oct 30 20:56:29.455211 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.455171 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-maas-api-tls\") pod \"85e81941-ca08-4234-9288-0dd4ecddf92f\" (UID: \"85e81941-ca08-4234-9288-0dd4ecddf92f\") " Oct 30 20:56:29.457080 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.456978 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e81941-ca08-4234-9288-0dd4ecddf92f-openshift-service-ca" pod "85e81941-ca08-4234-9288-0dd4ecddf92f" (UID: "85e81941-ca08-4234-9288-0dd4ecddf92f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:29.459564 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.459528 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-maas-api-tls" pod "85e81941-ca08-4234-9288-0dd4ecddf92f" (UID: "85e81941-ca08-4234-9288-0dd4ecddf92f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:56:29.460729 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.460699 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e81941-ca08-4234-9288-0dd4ecddf92f-kube-api-access-l28bw" pod "85e81941-ca08-4234-9288-0dd4ecddf92f" (UID: "85e81941-ca08-4234-9288-0dd4ecddf92f"). InnerVolumeSpecName "kube-api-access-l28bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:29.467809 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.467782 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-metrics-certs" pod "85e81941-ca08-4234-9288-0dd4ecddf92f" (UID: "85e81941-ca08-4234-9288-0dd4ecddf92f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:56:29.558333 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.558297 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-maas-api-tls\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:29.558574 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.558355 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l28bw\" (UniqueName: \"kubernetes.io/projected/85e81941-ca08-4234-9288-0dd4ecddf92f-kube-api-access-l28bw\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:29.558574 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.558371 5755 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e81941-ca08-4234-9288-0dd4ecddf92f-openshift-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:29.558574 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.558419 5755 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e81941-ca08-4234-9288-0dd4ecddf92f-metrics-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:29.902865 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.902831 5755 generic.go:356] "Generic (PLEG): container finished" podID="85e81941-ca08-4234-9288-0dd4ecddf92f" containerID="b645c249272cb0b0ee9b701cae19ccdace31539dec6c266f8192d3786391d2cc" exitCode=0 Oct 30 20:56:29.903319 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.902889 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" Oct 30 20:56:29.903319 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.902921 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" event={"ID":"85e81941-ca08-4234-9288-0dd4ecddf92f","Type":"ContainerDied","Data":"b645c249272cb0b0ee9b701cae19ccdace31539dec6c266f8192d3786391d2cc"} Oct 30 20:56:29.903319 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.903011 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42" event={"ID":"85e81941-ca08-4234-9288-0dd4ecddf92f","Type":"ContainerDied","Data":"8517b1ec09b22e4f6b3fa8bc7fda7837adc1cec6f15f8f0364512bad254976fe"} Oct 30 20:56:29.903319 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.903039 5755 scope.go:122] "RemoveContainer" containerID="b645c249272cb0b0ee9b701cae19ccdace31539dec6c266f8192d3786391d2cc" Oct 30 20:56:29.914298 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.914274 5755 scope.go:122] "RemoveContainer" containerID="b645c249272cb0b0ee9b701cae19ccdace31539dec6c266f8192d3786391d2cc" Oct 30 20:56:29.914580 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:56:29.914559 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b645c249272cb0b0ee9b701cae19ccdace31539dec6c266f8192d3786391d2cc\": container with ID starting with b645c249272cb0b0ee9b701cae19ccdace31539dec6c266f8192d3786391d2cc not found: ID does not exist" containerID="b645c249272cb0b0ee9b701cae19ccdace31539dec6c266f8192d3786391d2cc" Oct 30 20:56:29.914657 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:29.914593 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b645c249272cb0b0ee9b701cae19ccdace31539dec6c266f8192d3786391d2cc"} err="failed to get container status \"b645c249272cb0b0ee9b701cae19ccdace31539dec6c266f8192d3786391d2cc\": rpc error: code = NotFound desc = could not find container \"b645c249272cb0b0ee9b701cae19ccdace31539dec6c266f8192d3786391d2cc\": container with ID starting with b645c249272cb0b0ee9b701cae19ccdace31539dec6c266f8192d3786391d2cc not found: ID does not exist" Oct 30 20:56:30.074945 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:30.074905 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42"] Oct 30 20:56:30.137115 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:30.126485 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-86d620-566c586c9d-h7c42"] Oct 30 20:56:31.213095 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:31.213063 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e81941-ca08-4234-9288-0dd4ecddf92f" path="/var/lib/kubelet/pods/85e81941-ca08-4234-9288-0dd4ecddf92f/volumes" Oct 30 20:56:31.388773 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:31.388742 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw"] Oct 30 20:56:31.438085 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:31.438055 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-86d620-p2gmw"] Oct 30 20:56:33.212331 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:33.212299 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fd7656a-026c-4b86-a016-9e11446d27ae" path="/var/lib/kubelet/pods/7fd7656a-026c-4b86-a016-9e11446d27ae/volumes" Oct 30 20:56:38.914697 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:38.914612 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr"] Oct 30 20:56:38.915109 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:38.914806 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" podUID="56c12235-feb4-43a4-a2e3-32eb61584f28" containerName="istio-proxy" containerID="cri-o://2b7ce88e34380ab18cb6b287ea469dcd2d98050831c840d895233b501c8257e8" gracePeriod=30 Oct 30 20:56:38.932807 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:38.932777 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96"] Oct 30 20:56:38.933035 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:38.932991 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" podUID="e41e4f75-ea02-41f4-9698-c064cefa94ab" containerName="istio-proxy" containerID="cri-o://7bfc15a85c4b8764a870f683caf0ab9da019838b40582cf81156762ec6fb0c81" gracePeriod=30 Oct 30 20:56:39.768957 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:39.768915 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd"] Oct 30 20:56:39.769716 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:39.769699 5755 cpu_manager.go:414] "containerMap: removing container" podUID="85e81941-ca08-4234-9288-0dd4ecddf92f" containerName="maas-api" Oct 30 20:56:39.769795 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:39.769719 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e81941-ca08-4234-9288-0dd4ecddf92f" containerName="maas-api" Oct 30 20:56:39.769795 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:39.769731 5755 cpu_manager.go:414] "containerMap: removing container" podUID="7a5433df-a320-41cd-962f-240640c4af39" containerName="revoke-keys" Oct 30 20:56:39.769795 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:39.769736 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a5433df-a320-41cd-962f-240640c4af39" containerName="revoke-keys" Oct 30 20:56:39.769907 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:39.769836 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e81941-ca08-4234-9288-0dd4ecddf92f" containerName="maas-api" Oct 30 20:56:39.769907 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:39.769845 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a5433df-a320-41cd-962f-240640c4af39" containerName="revoke-keys" Oct 30 20:56:39.773340 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:39.773322 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd" Oct 30 20:56:39.829145 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:39.829115 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:56:39.833402 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:39.833352 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd"] Oct 30 20:56:39.877894 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:39.877864 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:56:39.931150 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:39.931110 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57fe592-756b-47a1-ac10-09836af845f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd\" (UID: \"c57fe592-756b-47a1-ac10-09836af845f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd" Oct 30 20:56:39.931585 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:39.931290 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28mwp\" (UniqueName: \"kubernetes.io/projected/c57fe592-756b-47a1-ac10-09836af845f4-kube-api-access-28mwp\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd\" (UID: \"c57fe592-756b-47a1-ac10-09836af845f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd" Oct 30 20:56:39.931585 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:39.931453 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57fe592-756b-47a1-ac10-09836af845f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd\" (UID: \"c57fe592-756b-47a1-ac10-09836af845f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd" Oct 30 20:56:40.032889 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:40.032790 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28mwp\" (UniqueName: \"kubernetes.io/projected/c57fe592-756b-47a1-ac10-09836af845f4-kube-api-access-28mwp\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd\" (UID: \"c57fe592-756b-47a1-ac10-09836af845f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd" Oct 30 20:56:40.033055 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:40.032916 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57fe592-756b-47a1-ac10-09836af845f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd\" (UID: \"c57fe592-756b-47a1-ac10-09836af845f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd" Oct 30 20:56:40.033055 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:40.032981 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57fe592-756b-47a1-ac10-09836af845f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd\" (UID: \"c57fe592-756b-47a1-ac10-09836af845f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd" Oct 30 20:56:40.033620 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:40.033599 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57fe592-756b-47a1-ac10-09836af845f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd\" (UID: \"c57fe592-756b-47a1-ac10-09836af845f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd" Oct 30 20:56:40.035500 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:40.035474 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57fe592-756b-47a1-ac10-09836af845f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd\" (UID: \"c57fe592-756b-47a1-ac10-09836af845f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd" Oct 30 20:56:40.098058 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:40.098026 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28mwp\" (UniqueName: \"kubernetes.io/projected/c57fe592-756b-47a1-ac10-09836af845f4-kube-api-access-28mwp\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd\" (UID: \"c57fe592-756b-47a1-ac10-09836af845f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd" Oct 30 20:56:40.386888 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:40.386810 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd" Oct 30 20:56:40.763894 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:40.763865 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd"] Oct 30 20:56:40.764512 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:56:40.764483 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57fe592_756b_47a1_ac10_09836af845f4.slice/crio-d0da0427e9441b855c8a7537965b07ded31b710f24665978003391b21fce2381 WatchSource:0}: Error finding container d0da0427e9441b855c8a7537965b07ded31b710f24665978003391b21fce2381: Status 404 returned error can't find the container with id d0da0427e9441b855c8a7537965b07ded31b710f24665978003391b21fce2381 Oct 30 20:56:40.981123 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:40.981089 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd" event={"ID":"c57fe592-756b-47a1-ac10-09836af845f4","Type":"ContainerStarted","Data":"d0da0427e9441b855c8a7537965b07ded31b710f24665978003391b21fce2381"} Oct 30 20:56:41.256689 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:56:41.256652 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:56:41.989977 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:41.989943 5755 generic.go:356] "Generic (PLEG): container finished" podID="c57fe592-756b-47a1-ac10-09836af845f4" containerID="d7837dc4a1cdd068b2801a2077ac52f472e1fed469861e34c6faaec4153f274c" exitCode=0 Oct 30 20:56:41.989977 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:41.989981 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd" event={"ID":"c57fe592-756b-47a1-ac10-09836af845f4","Type":"ContainerDied","Data":"d7837dc4a1cdd068b2801a2077ac52f472e1fed469861e34c6faaec4153f274c"} Oct 30 20:56:43.160977 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:43.160947 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd" Oct 30 20:56:43.279314 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:43.279234 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57fe592-756b-47a1-ac10-09836af845f4-kube-api-access-28mwp\" (UniqueName: \"kubernetes.io/projected/c57fe592-756b-47a1-ac10-09836af845f4-kube-api-access-28mwp\") pod \"c57fe592-756b-47a1-ac10-09836af845f4\" (UID: \"c57fe592-756b-47a1-ac10-09836af845f4\") " Oct 30 20:56:43.279501 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:43.279337 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c57fe592-756b-47a1-ac10-09836af845f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57fe592-756b-47a1-ac10-09836af845f4-maas-api-openshift-ca\") pod \"c57fe592-756b-47a1-ac10-09836af845f4\" (UID: \"c57fe592-756b-47a1-ac10-09836af845f4\") " Oct 30 20:56:43.279501 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:43.279371 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c57fe592-756b-47a1-ac10-09836af845f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57fe592-756b-47a1-ac10-09836af845f4-maas-api-xks-ca\") pod \"c57fe592-756b-47a1-ac10-09836af845f4\" (UID: \"c57fe592-756b-47a1-ac10-09836af845f4\") " Oct 30 20:56:43.279721 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:43.279700 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57fe592-756b-47a1-ac10-09836af845f4-maas-api-openshift-ca" pod "c57fe592-756b-47a1-ac10-09836af845f4" (UID: "c57fe592-756b-47a1-ac10-09836af845f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:43.279876 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:43.279862 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57fe592-756b-47a1-ac10-09836af845f4-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:43.281637 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:43.281610 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57fe592-756b-47a1-ac10-09836af845f4-maas-api-xks-ca" pod "c57fe592-756b-47a1-ac10-09836af845f4" (UID: "c57fe592-756b-47a1-ac10-09836af845f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:56:43.281779 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:43.281760 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57fe592-756b-47a1-ac10-09836af845f4-kube-api-access-28mwp" pod "c57fe592-756b-47a1-ac10-09836af845f4" (UID: "c57fe592-756b-47a1-ac10-09836af845f4"). InnerVolumeSpecName "kube-api-access-28mwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:43.381688 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:43.381651 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28mwp\" (UniqueName: \"kubernetes.io/projected/c57fe592-756b-47a1-ac10-09836af845f4-kube-api-access-28mwp\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:43.381688 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:43.381684 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57fe592-756b-47a1-ac10-09836af845f4-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.004446 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.004414 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd" event={"ID":"c57fe592-756b-47a1-ac10-09836af845f4","Type":"ContainerDied","Data":"d0da0427e9441b855c8a7537965b07ded31b710f24665978003391b21fce2381"} Oct 30 20:56:44.004557 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.004447 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd" Oct 30 20:56:44.004557 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.004470 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0da0427e9441b855c8a7537965b07ded31b710f24665978003391b21fce2381" Oct 30 20:56:44.204074 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.204049 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:56:44.253077 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.253049 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:56:44.295203 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.295174 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-envoy\") pod \"56c12235-feb4-43a4-a2e3-32eb61584f28\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " Oct 30 20:56:44.295366 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.295238 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-podinfo\") pod \"56c12235-feb4-43a4-a2e3-32eb61584f28\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " Oct 30 20:56:44.295366 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.295278 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-token\" (UniqueName: \"kubernetes.io/projected/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-token\") pod \"56c12235-feb4-43a4-a2e3-32eb61584f28\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " Oct 30 20:56:44.295366 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.295307 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-credential-socket\") pod \"56c12235-feb4-43a4-a2e3-32eb61584f28\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " Oct 30 20:56:44.295550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.295370 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-data\") pod \"56c12235-feb4-43a4-a2e3-32eb61584f28\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " Oct 30 20:56:44.295550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.295433 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-workload-certs\") pod \"56c12235-feb4-43a4-a2e3-32eb61584f28\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " Oct 30 20:56:44.295550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.295508 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c12235-feb4-43a4-a2e3-32eb61584f28-kube-api-access-zl4l5\" (UniqueName: \"kubernetes.io/projected/56c12235-feb4-43a4-a2e3-32eb61584f28-kube-api-access-zl4l5\") pod \"56c12235-feb4-43a4-a2e3-32eb61584f28\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " Oct 30 20:56:44.295550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.295539 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56c12235-feb4-43a4-a2e3-32eb61584f28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c12235-feb4-43a4-a2e3-32eb61584f28-istiod-ca-cert\") pod \"56c12235-feb4-43a4-a2e3-32eb61584f28\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " Oct 30 20:56:44.295765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.295620 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-workload-socket\") pod \"56c12235-feb4-43a4-a2e3-32eb61584f28\" (UID: \"56c12235-feb4-43a4-a2e3-32eb61584f28\") " Oct 30 20:56:44.295765 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.295733 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-data" pod "56c12235-feb4-43a4-a2e3-32eb61584f28" (UID: "56c12235-feb4-43a4-a2e3-32eb61584f28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:56:44.295859 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.295838 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-credential-socket" pod "56c12235-feb4-43a4-a2e3-32eb61584f28" (UID: "56c12235-feb4-43a4-a2e3-32eb61584f28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:56:44.295913 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.295855 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-workload-certs" pod "56c12235-feb4-43a4-a2e3-32eb61584f28" (UID: "56c12235-feb4-43a4-a2e3-32eb61584f28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:56:44.296187 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.296117 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c12235-feb4-43a4-a2e3-32eb61584f28-istiod-ca-cert" pod "56c12235-feb4-43a4-a2e3-32eb61584f28" (UID: "56c12235-feb4-43a4-a2e3-32eb61584f28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:44.296255 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.296206 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-workload-socket" pod "56c12235-feb4-43a4-a2e3-32eb61584f28" (UID: "56c12235-feb4-43a4-a2e3-32eb61584f28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:56:44.296751 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.296725 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.296853 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.296755 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.296853 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.296772 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.296853 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.296785 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.296853 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.296798 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c12235-feb4-43a4-a2e3-32eb61584f28-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.297775 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.297748 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-token" pod "56c12235-feb4-43a4-a2e3-32eb61584f28" (UID: "56c12235-feb4-43a4-a2e3-32eb61584f28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:44.297882 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.297801 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-podinfo" pod "56c12235-feb4-43a4-a2e3-32eb61584f28" (UID: "56c12235-feb4-43a4-a2e3-32eb61584f28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:56:44.297882 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.297802 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-envoy" pod "56c12235-feb4-43a4-a2e3-32eb61584f28" (UID: "56c12235-feb4-43a4-a2e3-32eb61584f28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:56:44.297882 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.297844 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c12235-feb4-43a4-a2e3-32eb61584f28-kube-api-access-zl4l5" pod "56c12235-feb4-43a4-a2e3-32eb61584f28" (UID: "56c12235-feb4-43a4-a2e3-32eb61584f28"). InnerVolumeSpecName "kube-api-access-zl4l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:44.397977 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.397947 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-token\" (UniqueName: \"kubernetes.io/projected/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-token\") pod \"e41e4f75-ea02-41f4-9698-c064cefa94ab\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " Oct 30 20:56:44.398170 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.398059 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-envoy\") pod \"e41e4f75-ea02-41f4-9698-c064cefa94ab\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " Oct 30 20:56:44.398170 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.398147 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-data\") pod \"e41e4f75-ea02-41f4-9698-c064cefa94ab\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " Oct 30 20:56:44.398299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.398240 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-podinfo\") pod \"e41e4f75-ea02-41f4-9698-c064cefa94ab\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " Oct 30 20:56:44.398299 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.398274 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-workload-certs\") pod \"e41e4f75-ea02-41f4-9698-c064cefa94ab\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " Oct 30 20:56:44.398441 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.398342 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e41e4f75-ea02-41f4-9698-c064cefa94ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41e4f75-ea02-41f4-9698-c064cefa94ab-istiod-ca-cert\") pod \"e41e4f75-ea02-41f4-9698-c064cefa94ab\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " Oct 30 20:56:44.398441 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.398373 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-credential-socket\") pod \"e41e4f75-ea02-41f4-9698-c064cefa94ab\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " Oct 30 20:56:44.398441 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.398431 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-workload-socket\") pod \"e41e4f75-ea02-41f4-9698-c064cefa94ab\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " Oct 30 20:56:44.398625 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.398504 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41e4f75-ea02-41f4-9698-c064cefa94ab-kube-api-access-42lmw\" (UniqueName: \"kubernetes.io/projected/e41e4f75-ea02-41f4-9698-c064cefa94ab-kube-api-access-42lmw\") pod \"e41e4f75-ea02-41f4-9698-c064cefa94ab\" (UID: \"e41e4f75-ea02-41f4-9698-c064cefa94ab\") " Oct 30 20:56:44.399559 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.399019 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41e4f75-ea02-41f4-9698-c064cefa94ab-istiod-ca-cert" pod "e41e4f75-ea02-41f4-9698-c064cefa94ab" (UID: "e41e4f75-ea02-41f4-9698-c064cefa94ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:44.399559 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.399212 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41e4f75-ea02-41f4-9698-c064cefa94ab-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.399559 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.399234 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl4l5\" (UniqueName: \"kubernetes.io/projected/56c12235-feb4-43a4-a2e3-32eb61584f28-kube-api-access-zl4l5\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.399559 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.399250 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.399559 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.399262 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.399559 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.399275 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c12235-feb4-43a4-a2e3-32eb61584f28-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.399559 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.399455 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-data" pod "e41e4f75-ea02-41f4-9698-c064cefa94ab" (UID: "e41e4f75-ea02-41f4-9698-c064cefa94ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:56:44.401254 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.401193 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-token" pod "e41e4f75-ea02-41f4-9698-c064cefa94ab" (UID: "e41e4f75-ea02-41f4-9698-c064cefa94ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:44.401509 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.401463 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-workload-socket" pod "e41e4f75-ea02-41f4-9698-c064cefa94ab" (UID: "e41e4f75-ea02-41f4-9698-c064cefa94ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:56:44.401625 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.401591 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-workload-certs" pod "e41e4f75-ea02-41f4-9698-c064cefa94ab" (UID: "e41e4f75-ea02-41f4-9698-c064cefa94ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:56:44.403517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.403489 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-credential-socket" pod "e41e4f75-ea02-41f4-9698-c064cefa94ab" (UID: "e41e4f75-ea02-41f4-9698-c064cefa94ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:56:44.404980 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.404957 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41e4f75-ea02-41f4-9698-c064cefa94ab-kube-api-access-42lmw" pod "e41e4f75-ea02-41f4-9698-c064cefa94ab" (UID: "e41e4f75-ea02-41f4-9698-c064cefa94ab"). InnerVolumeSpecName "kube-api-access-42lmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:44.404980 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.404971 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-podinfo" pod "e41e4f75-ea02-41f4-9698-c064cefa94ab" (UID: "e41e4f75-ea02-41f4-9698-c064cefa94ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:56:44.405137 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.405034 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-envoy" pod "e41e4f75-ea02-41f4-9698-c064cefa94ab" (UID: "e41e4f75-ea02-41f4-9698-c064cefa94ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:56:44.501023 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.500993 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42lmw\" (UniqueName: \"kubernetes.io/projected/e41e4f75-ea02-41f4-9698-c064cefa94ab-kube-api-access-42lmw\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.501023 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.501020 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.501023 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.501031 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.501260 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.501039 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.501260 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.501047 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41e4f75-ea02-41f4-9698-c064cefa94ab-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.501260 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.501057 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.501260 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.501065 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:44.501260 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:44.501075 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41e4f75-ea02-41f4-9698-c064cefa94ab-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:45.014066 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.014030 5755 generic.go:356] "Generic (PLEG): container finished" podID="e41e4f75-ea02-41f4-9698-c064cefa94ab" containerID="7bfc15a85c4b8764a870f683caf0ab9da019838b40582cf81156762ec6fb0c81" exitCode=0 Oct 30 20:56:45.014244 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.014081 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" Oct 30 20:56:45.014244 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.014113 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" event={"ID":"e41e4f75-ea02-41f4-9698-c064cefa94ab","Type":"ContainerDied","Data":"7bfc15a85c4b8764a870f683caf0ab9da019838b40582cf81156762ec6fb0c81"} Oct 30 20:56:45.014244 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.014161 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96" event={"ID":"e41e4f75-ea02-41f4-9698-c064cefa94ab","Type":"ContainerDied","Data":"77aa3fcdd1fc77f28ec178b27f62e5f2e2ef1db1fcbc8e413f961ac9d4eba27a"} Oct 30 20:56:45.014244 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.014182 5755 scope.go:122] "RemoveContainer" containerID="7bfc15a85c4b8764a870f683caf0ab9da019838b40582cf81156762ec6fb0c81" Oct 30 20:56:45.015767 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.015696 5755 generic.go:356] "Generic (PLEG): container finished" podID="56c12235-feb4-43a4-a2e3-32eb61584f28" containerID="2b7ce88e34380ab18cb6b287ea469dcd2d98050831c840d895233b501c8257e8" exitCode=0 Oct 30 20:56:45.015767 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.015741 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" Oct 30 20:56:45.015767 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.015780 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" event={"ID":"56c12235-feb4-43a4-a2e3-32eb61584f28","Type":"ContainerDied","Data":"2b7ce88e34380ab18cb6b287ea469dcd2d98050831c840d895233b501c8257e8"} Oct 30 20:56:45.016132 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.015835 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr" event={"ID":"56c12235-feb4-43a4-a2e3-32eb61584f28","Type":"ContainerDied","Data":"8094d6dd692770ac61c3a6308062a907cd6d5f136e218555e506855740302e7c"} Oct 30 20:56:45.030094 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.029693 5755 scope.go:122] "RemoveContainer" containerID="7bfc15a85c4b8764a870f683caf0ab9da019838b40582cf81156762ec6fb0c81" Oct 30 20:56:45.030094 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:56:45.030062 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bfc15a85c4b8764a870f683caf0ab9da019838b40582cf81156762ec6fb0c81\": container with ID starting with 7bfc15a85c4b8764a870f683caf0ab9da019838b40582cf81156762ec6fb0c81 not found: ID does not exist" containerID="7bfc15a85c4b8764a870f683caf0ab9da019838b40582cf81156762ec6fb0c81" Oct 30 20:56:45.030231 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.030093 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bfc15a85c4b8764a870f683caf0ab9da019838b40582cf81156762ec6fb0c81"} err="failed to get container status \"7bfc15a85c4b8764a870f683caf0ab9da019838b40582cf81156762ec6fb0c81\": rpc error: code = NotFound desc = could not find container \"7bfc15a85c4b8764a870f683caf0ab9da019838b40582cf81156762ec6fb0c81\": container with ID starting with 7bfc15a85c4b8764a870f683caf0ab9da019838b40582cf81156762ec6fb0c81 not found: ID does not exist" Oct 30 20:56:45.030231 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.030114 5755 scope.go:122] "RemoveContainer" containerID="2b7ce88e34380ab18cb6b287ea469dcd2d98050831c840d895233b501c8257e8" Oct 30 20:56:45.040542 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.040237 5755 scope.go:122] "RemoveContainer" containerID="2b7ce88e34380ab18cb6b287ea469dcd2d98050831c840d895233b501c8257e8" Oct 30 20:56:45.040698 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:56:45.040664 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b7ce88e34380ab18cb6b287ea469dcd2d98050831c840d895233b501c8257e8\": container with ID starting with 2b7ce88e34380ab18cb6b287ea469dcd2d98050831c840d895233b501c8257e8 not found: ID does not exist" containerID="2b7ce88e34380ab18cb6b287ea469dcd2d98050831c840d895233b501c8257e8" Oct 30 20:56:45.040779 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.040703 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7ce88e34380ab18cb6b287ea469dcd2d98050831c840d895233b501c8257e8"} err="failed to get container status \"2b7ce88e34380ab18cb6b287ea469dcd2d98050831c840d895233b501c8257e8\": rpc error: code = NotFound desc = could not find container \"2b7ce88e34380ab18cb6b287ea469dcd2d98050831c840d895233b501c8257e8\": container with ID starting with 2b7ce88e34380ab18cb6b287ea469dcd2d98050831c840d895233b501c8257e8 not found: ID does not exist" Oct 30 20:56:45.182005 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.181975 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr"] Oct 30 20:56:45.186496 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.186473 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-7psnr"] Oct 30 20:56:45.213951 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.213924 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c12235-feb4-43a4-a2e3-32eb61584f28" path="/var/lib/kubelet/pods/56c12235-feb4-43a4-a2e3-32eb61584f28/volumes" Oct 30 20:56:45.299138 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.299062 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96"] Oct 30 20:56:45.303244 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:45.303220 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hvm96"] Oct 30 20:56:47.213122 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:47.213090 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e41e4f75-ea02-41f4-9698-c064cefa94ab" path="/var/lib/kubelet/pods/e41e4f75-ea02-41f4-9698-c064cefa94ab/volumes" Oct 30 20:56:54.712995 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:54.712958 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv"] Oct 30 20:56:54.762563 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:54.762525 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b86743-47vkv"] Oct 30 20:56:55.213562 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:56:55.213530 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a5433df-a320-41cd-962f-240640c4af39" path="/var/lib/kubelet/pods/7a5433df-a320-41cd-962f-240640c4af39/volumes" Oct 30 20:57:03.246555 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.246521 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9"] Oct 30 20:57:03.247066 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.247051 5755 cpu_manager.go:414] "containerMap: removing container" podUID="c57fe592-756b-47a1-ac10-09836af845f4" containerName="revoke-keys" Oct 30 20:57:03.247117 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.247068 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57fe592-756b-47a1-ac10-09836af845f4" containerName="revoke-keys" Oct 30 20:57:03.247117 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.247106 5755 cpu_manager.go:414] "containerMap: removing container" podUID="56c12235-feb4-43a4-a2e3-32eb61584f28" containerName="istio-proxy" Oct 30 20:57:03.247117 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.247110 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c12235-feb4-43a4-a2e3-32eb61584f28" containerName="istio-proxy" Oct 30 20:57:03.247209 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.247120 5755 cpu_manager.go:414] "containerMap: removing container" podUID="e41e4f75-ea02-41f4-9698-c064cefa94ab" containerName="istio-proxy" Oct 30 20:57:03.247209 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.247125 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41e4f75-ea02-41f4-9698-c064cefa94ab" containerName="istio-proxy" Oct 30 20:57:03.247273 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.247220 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c12235-feb4-43a4-a2e3-32eb61584f28" containerName="istio-proxy" Oct 30 20:57:03.247273 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.247229 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41e4f75-ea02-41f4-9698-c064cefa94ab" containerName="istio-proxy" Oct 30 20:57:03.247273 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.247238 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57fe592-756b-47a1-ac10-09836af845f4" containerName="revoke-keys" Oct 30 20:57:03.252211 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.252190 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9" Oct 30 20:57:03.307149 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.307109 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9"] Oct 30 20:57:03.352264 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.352234 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:57:03.352452 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.352311 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:57:03.424285 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.424255 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6tcq\" (UniqueName: \"kubernetes.io/projected/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-kube-api-access-k6tcq\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9\" (UID: \"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9" Oct 30 20:57:03.424522 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.424370 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9\" (UID: \"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9" Oct 30 20:57:03.424522 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.424487 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9\" (UID: \"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9" Oct 30 20:57:03.526126 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.526032 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9\" (UID: \"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9" Oct 30 20:57:03.526304 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.526216 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6tcq\" (UniqueName: \"kubernetes.io/projected/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-kube-api-access-k6tcq\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9\" (UID: \"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9" Oct 30 20:57:03.526417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.526306 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9\" (UID: \"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9" Oct 30 20:57:03.526750 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.526730 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9\" (UID: \"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9" Oct 30 20:57:03.528714 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.528692 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9\" (UID: \"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9" Oct 30 20:57:03.584662 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.584633 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6tcq\" (UniqueName: \"kubernetes.io/projected/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-kube-api-access-k6tcq\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9\" (UID: \"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9" Oct 30 20:57:03.865230 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:03.865143 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9" Oct 30 20:57:04.233208 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:04.232309 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b"] Oct 30 20:57:04.233208 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:04.232565 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" podUID="80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" containerName="istio-proxy" containerID="cri-o://9cda15cfe7c97261babd672c204d36facc1f1b977bc65661a298e2eb33529ca8" gracePeriod=30 Oct 30 20:57:04.280417 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:04.280356 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp"] Oct 30 20:57:04.280782 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:04.280604 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" podUID="1e3d6326-166f-47f0-888e-f7a5e8464aa1" containerName="istio-proxy" containerID="cri-o://696e390e9ea0c046828e9c8dbfedc266101bd78975b79f587da363163bd32f94" gracePeriod=30 Oct 30 20:57:04.284228 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:04.284197 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9"] Oct 30 20:57:04.290600 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:57:04.290566 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a7ae95_219f_4ed4_a8df_2fd2ef61d9f3.slice/crio-dbc9f564434a66c18b10c3ce67ee8b7af95544274758cb7b9a613aaec8a6cf28 WatchSource:0}: Error finding container dbc9f564434a66c18b10c3ce67ee8b7af95544274758cb7b9a613aaec8a6cf28: Status 404 returned error can't find the container with id dbc9f564434a66c18b10c3ce67ee8b7af95544274758cb7b9a613aaec8a6cf28 Oct 30 20:57:05.248642 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:05.248604 5755 generic.go:356] "Generic (PLEG): container finished" podID="70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3" containerID="b4e553004cba8e970702cfaf40d2c8a014c3577938883782ebfbd119a5be94ab" exitCode=0 Oct 30 20:57:05.248837 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:05.248685 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9" event={"ID":"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3","Type":"ContainerDied","Data":"b4e553004cba8e970702cfaf40d2c8a014c3577938883782ebfbd119a5be94ab"} Oct 30 20:57:05.248837 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:05.248743 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9" event={"ID":"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3","Type":"ContainerStarted","Data":"dbc9f564434a66c18b10c3ce67ee8b7af95544274758cb7b9a613aaec8a6cf28"} Oct 30 20:57:06.057035 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.057003 5755 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:57:06.057450 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.057059 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" podUID="80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:57:06.217726 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.217690 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:57:06.217884 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.217756 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" podUID="1e3d6326-166f-47f0-888e-f7a5e8464aa1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:57:06.325493 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.325460 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx"] Oct 30 20:57:06.329477 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.329453 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx" Oct 30 20:57:06.387998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.387970 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx"] Oct 30 20:57:06.409715 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.409685 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9" Oct 30 20:57:06.463209 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.463175 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0906f40-8902-4c30-b0ea-7576edea2f8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c83a089-9fflx\" (UID: \"b0906f40-8902-4c30-b0ea-7576edea2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx" Oct 30 20:57:06.463379 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.463360 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0906f40-8902-4c30-b0ea-7576edea2f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c83a089-9fflx\" (UID: \"b0906f40-8902-4c30-b0ea-7576edea2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx" Oct 30 20:57:06.463496 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.463481 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjwdh\" (UniqueName: \"kubernetes.io/projected/b0906f40-8902-4c30-b0ea-7576edea2f8c-kube-api-access-sjwdh\") pod \"maas-api-revoke-keys-e2e-mt-0c83a089-9fflx\" (UID: \"b0906f40-8902-4c30-b0ea-7576edea2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx" Oct 30 20:57:06.564952 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.564868 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-kube-api-access-k6tcq\" (UniqueName: \"kubernetes.io/projected/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-kube-api-access-k6tcq\") pod \"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3\" (UID: \"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3\") " Oct 30 20:57:06.565128 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.565047 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-maas-api-openshift-ca\") pod \"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3\" (UID: \"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3\") " Oct 30 20:57:06.565189 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.565152 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-maas-api-xks-ca\") pod \"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3\" (UID: \"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3\") " Oct 30 20:57:06.565489 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.565462 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0906f40-8902-4c30-b0ea-7576edea2f8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c83a089-9fflx\" (UID: \"b0906f40-8902-4c30-b0ea-7576edea2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx" Oct 30 20:57:06.565850 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.565827 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0906f40-8902-4c30-b0ea-7576edea2f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c83a089-9fflx\" (UID: \"b0906f40-8902-4c30-b0ea-7576edea2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx" Oct 30 20:57:06.566034 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.566017 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjwdh\" (UniqueName: \"kubernetes.io/projected/b0906f40-8902-4c30-b0ea-7576edea2f8c-kube-api-access-sjwdh\") pod \"maas-api-revoke-keys-e2e-mt-0c83a089-9fflx\" (UID: \"b0906f40-8902-4c30-b0ea-7576edea2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx" Oct 30 20:57:06.567461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.567413 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0906f40-8902-4c30-b0ea-7576edea2f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c83a089-9fflx\" (UID: \"b0906f40-8902-4c30-b0ea-7576edea2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx" Oct 30 20:57:06.567571 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.565606 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-maas-api-openshift-ca" pod "70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3" (UID: "70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:06.567671 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.567543 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-kube-api-access-k6tcq" pod "70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3" (UID: "70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3"). InnerVolumeSpecName "kube-api-access-k6tcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:06.568463 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.568440 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-maas-api-xks-ca" pod "70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3" (UID: "70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:06.568945 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.568926 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0906f40-8902-4c30-b0ea-7576edea2f8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c83a089-9fflx\" (UID: \"b0906f40-8902-4c30-b0ea-7576edea2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx" Oct 30 20:57:06.624937 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.624906 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjwdh\" (UniqueName: \"kubernetes.io/projected/b0906f40-8902-4c30-b0ea-7576edea2f8c-kube-api-access-sjwdh\") pod \"maas-api-revoke-keys-e2e-mt-0c83a089-9fflx\" (UID: \"b0906f40-8902-4c30-b0ea-7576edea2f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx" Oct 30 20:57:06.645919 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.645888 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx" Oct 30 20:57:06.668170 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.668135 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:06.668288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.668175 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6tcq\" (UniqueName: \"kubernetes.io/projected/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-kube-api-access-k6tcq\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:06.668288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:06.668192 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:07.033043 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:07.032996 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx"] Oct 30 20:57:07.040440 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:57:07.040401 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0906f40_8902_4c30_b0ea_7576edea2f8c.slice/crio-ee155f8e69e82d5171376d60e632df979ce85c34754cbff12bf37848200eeb34 WatchSource:0}: Error finding container ee155f8e69e82d5171376d60e632df979ce85c34754cbff12bf37848200eeb34: Status 404 returned error can't find the container with id ee155f8e69e82d5171376d60e632df979ce85c34754cbff12bf37848200eeb34 Oct 30 20:57:07.264205 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:07.264163 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx" event={"ID":"b0906f40-8902-4c30-b0ea-7576edea2f8c","Type":"ContainerStarted","Data":"ee155f8e69e82d5171376d60e632df979ce85c34754cbff12bf37848200eeb34"} Oct 30 20:57:07.265630 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:07.265611 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9" Oct 30 20:57:07.265729 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:07.265651 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9" event={"ID":"70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3","Type":"ContainerDied","Data":"dbc9f564434a66c18b10c3ce67ee8b7af95544274758cb7b9a613aaec8a6cf28"} Oct 30 20:57:07.265729 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:07.265704 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbc9f564434a66c18b10c3ce67ee8b7af95544274758cb7b9a613aaec8a6cf28" Oct 30 20:57:08.274903 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:08.274866 5755 generic.go:356] "Generic (PLEG): container finished" podID="b0906f40-8902-4c30-b0ea-7576edea2f8c" containerID="27d3b77051b01c62446a2d5a05a5412e1c09d21d98778036a0580ac3b7423818" exitCode=0 Oct 30 20:57:08.275295 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:08.274917 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx" event={"ID":"b0906f40-8902-4c30-b0ea-7576edea2f8c","Type":"ContainerDied","Data":"27d3b77051b01c62446a2d5a05a5412e1c09d21d98778036a0580ac3b7423818"} Oct 30 20:57:09.481739 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.481210 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx" Oct 30 20:57:09.517808 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.517012 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0906f40-8902-4c30-b0ea-7576edea2f8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0906f40-8902-4c30-b0ea-7576edea2f8c-maas-api-xks-ca\") pod \"b0906f40-8902-4c30-b0ea-7576edea2f8c\" (UID: \"b0906f40-8902-4c30-b0ea-7576edea2f8c\") " Oct 30 20:57:09.517808 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.517074 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0906f40-8902-4c30-b0ea-7576edea2f8c-kube-api-access-sjwdh\" (UniqueName: \"kubernetes.io/projected/b0906f40-8902-4c30-b0ea-7576edea2f8c-kube-api-access-sjwdh\") pod \"b0906f40-8902-4c30-b0ea-7576edea2f8c\" (UID: \"b0906f40-8902-4c30-b0ea-7576edea2f8c\") " Oct 30 20:57:09.517808 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.517170 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0906f40-8902-4c30-b0ea-7576edea2f8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0906f40-8902-4c30-b0ea-7576edea2f8c-maas-api-openshift-ca\") pod \"b0906f40-8902-4c30-b0ea-7576edea2f8c\" (UID: \"b0906f40-8902-4c30-b0ea-7576edea2f8c\") " Oct 30 20:57:09.518319 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.518282 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0906f40-8902-4c30-b0ea-7576edea2f8c-maas-api-openshift-ca" pod "b0906f40-8902-4c30-b0ea-7576edea2f8c" (UID: "b0906f40-8902-4c30-b0ea-7576edea2f8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:09.518454 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.518439 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0906f40-8902-4c30-b0ea-7576edea2f8c-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.525053 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.525025 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0906f40-8902-4c30-b0ea-7576edea2f8c-kube-api-access-sjwdh" pod "b0906f40-8902-4c30-b0ea-7576edea2f8c" (UID: "b0906f40-8902-4c30-b0ea-7576edea2f8c"). InnerVolumeSpecName "kube-api-access-sjwdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:09.525224 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.525204 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0906f40-8902-4c30-b0ea-7576edea2f8c-maas-api-xks-ca" pod "b0906f40-8902-4c30-b0ea-7576edea2f8c" (UID: "b0906f40-8902-4c30-b0ea-7576edea2f8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:09.543526 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.543504 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:57:09.588463 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.588439 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:57:09.619160 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619127 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istiod-ca-cert\") pod \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " Oct 30 20:57:09.619324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619180 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-workload-socket\") pod \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " Oct 30 20:57:09.619324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619238 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-podinfo\") pod \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " Oct 30 20:57:09.619324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619274 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3d6326-166f-47f0-888e-f7a5e8464aa1-kube-api-access-8ds42\" (UniqueName: \"kubernetes.io/projected/1e3d6326-166f-47f0-888e-f7a5e8464aa1-kube-api-access-8ds42\") pod \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " Oct 30 20:57:09.619619 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619365 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-podinfo\") pod \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " Oct 30 20:57:09.619619 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619474 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-credential-socket\") pod \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " Oct 30 20:57:09.619619 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619517 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-envoy\") pod \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " Oct 30 20:57:09.619619 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619582 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istiod-ca-cert\") pod \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " Oct 30 20:57:09.619619 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619616 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-workload-certs\") pod \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " Oct 30 20:57:09.619867 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619644 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-credential-socket\") pod \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " Oct 30 20:57:09.619867 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619680 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-token\" (UniqueName: \"kubernetes.io/projected/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-token\") pod \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " Oct 30 20:57:09.619867 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619711 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-workload-certs\") pod \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " Oct 30 20:57:09.619867 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619784 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-data\") pod \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " Oct 30 20:57:09.619867 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619819 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-token\" (UniqueName: \"kubernetes.io/projected/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-token\") pod \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " Oct 30 20:57:09.619867 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619851 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-envoy\") pod \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " Oct 30 20:57:09.620405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619884 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-kube-api-access-cjn7l\" (UniqueName: \"kubernetes.io/projected/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-kube-api-access-cjn7l\") pod \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " Oct 30 20:57:09.620405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619577 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-workload-socket" pod "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" (UID: "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:09.620405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619977 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-data\") pod \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\" (UID: \"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a\") " Oct 30 20:57:09.620405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619587 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istiod-ca-cert" pod "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" (UID: "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:09.620405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619763 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-credential-socket" pod "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" (UID: "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:09.620405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.619960 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-workload-certs" pod "1e3d6326-166f-47f0-888e-f7a5e8464aa1" (UID: "1e3d6326-166f-47f0-888e-f7a5e8464aa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:09.620405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.620012 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-workload-socket\") pod \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\" (UID: \"1e3d6326-166f-47f0-888e-f7a5e8464aa1\") " Oct 30 20:57:09.620405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.620168 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-data" pod "1e3d6326-166f-47f0-888e-f7a5e8464aa1" (UID: "1e3d6326-166f-47f0-888e-f7a5e8464aa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:09.620405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.620235 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istiod-ca-cert" pod "1e3d6326-166f-47f0-888e-f7a5e8464aa1" (UID: "1e3d6326-166f-47f0-888e-f7a5e8464aa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:09.620405 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.620379 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-credential-socket" pod "1e3d6326-166f-47f0-888e-f7a5e8464aa1" (UID: "1e3d6326-166f-47f0-888e-f7a5e8464aa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:09.620922 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.620569 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-workload-certs" pod "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" (UID: "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:09.620922 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.620723 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-data" pod "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" (UID: "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:09.621024 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.621001 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.621085 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.621033 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.621085 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.621049 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.621085 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.621063 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.621085 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.621077 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.621274 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.621091 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.621274 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.621104 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.621274 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.621117 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.621274 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.621131 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.621274 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.621144 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0906f40-8902-4c30-b0ea-7576edea2f8c-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.621274 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.621159 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjwdh\" (UniqueName: \"kubernetes.io/projected/b0906f40-8902-4c30-b0ea-7576edea2f8c-kube-api-access-sjwdh\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.621621 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.621357 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-workload-socket" pod "1e3d6326-166f-47f0-888e-f7a5e8464aa1" (UID: "1e3d6326-166f-47f0-888e-f7a5e8464aa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:09.622359 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.622314 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-envoy" pod "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" (UID: "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:09.623748 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.623700 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-kube-api-access-cjn7l" pod "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" (UID: "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a"). InnerVolumeSpecName "kube-api-access-cjn7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:09.623849 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.623741 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-podinfo" pod "1e3d6326-166f-47f0-888e-f7a5e8464aa1" (UID: "1e3d6326-166f-47f0-888e-f7a5e8464aa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:09.624034 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.624001 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-podinfo" pod "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" (UID: "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:09.624202 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.624175 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-envoy" pod "1e3d6326-166f-47f0-888e-f7a5e8464aa1" (UID: "1e3d6326-166f-47f0-888e-f7a5e8464aa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:09.624768 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.624749 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3d6326-166f-47f0-888e-f7a5e8464aa1-kube-api-access-8ds42" pod "1e3d6326-166f-47f0-888e-f7a5e8464aa1" (UID: "1e3d6326-166f-47f0-888e-f7a5e8464aa1"). InnerVolumeSpecName "kube-api-access-8ds42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:09.625192 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.625174 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-token" pod "1e3d6326-166f-47f0-888e-f7a5e8464aa1" (UID: "1e3d6326-166f-47f0-888e-f7a5e8464aa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:09.625496 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.625477 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-token" pod "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" (UID: "80aa1317-6c2c-48b4-a5f5-bbfa7928f08a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:09.722010 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.721978 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.722010 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.722008 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ds42\" (UniqueName: \"kubernetes.io/projected/1e3d6326-166f-47f0-888e-f7a5e8464aa1-kube-api-access-8ds42\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.722010 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.722018 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.722240 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.722027 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.722240 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.722036 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.722240 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.722044 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.722240 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.722051 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.722240 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.722059 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjn7l\" (UniqueName: \"kubernetes.io/projected/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a-kube-api-access-cjn7l\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:09.722240 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:09.722067 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e3d6326-166f-47f0-888e-f7a5e8464aa1-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:10.290967 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.290929 5755 generic.go:356] "Generic (PLEG): container finished" podID="80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" containerID="9cda15cfe7c97261babd672c204d36facc1f1b977bc65661a298e2eb33529ca8" exitCode=0 Oct 30 20:57:10.291138 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.291011 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" Oct 30 20:57:10.291138 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.291007 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" event={"ID":"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a","Type":"ContainerDied","Data":"9cda15cfe7c97261babd672c204d36facc1f1b977bc65661a298e2eb33529ca8"} Oct 30 20:57:10.291138 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.291129 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b" event={"ID":"80aa1317-6c2c-48b4-a5f5-bbfa7928f08a","Type":"ContainerDied","Data":"26f2941eaf04c3a2800d4e1d15196db626c47e6d24e9ac4c20187f7741934885"} Oct 30 20:57:10.291281 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.291158 5755 scope.go:122] "RemoveContainer" containerID="9cda15cfe7c97261babd672c204d36facc1f1b977bc65661a298e2eb33529ca8" Oct 30 20:57:10.292705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.292516 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx" Oct 30 20:57:10.292705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.292545 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx" event={"ID":"b0906f40-8902-4c30-b0ea-7576edea2f8c","Type":"ContainerDied","Data":"ee155f8e69e82d5171376d60e632df979ce85c34754cbff12bf37848200eeb34"} Oct 30 20:57:10.292705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.292585 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee155f8e69e82d5171376d60e632df979ce85c34754cbff12bf37848200eeb34" Oct 30 20:57:10.294328 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.294301 5755 generic.go:356] "Generic (PLEG): container finished" podID="1e3d6326-166f-47f0-888e-f7a5e8464aa1" containerID="696e390e9ea0c046828e9c8dbfedc266101bd78975b79f587da363163bd32f94" exitCode=0 Oct 30 20:57:10.294487 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.294350 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" Oct 30 20:57:10.294487 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.294415 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" event={"ID":"1e3d6326-166f-47f0-888e-f7a5e8464aa1","Type":"ContainerDied","Data":"696e390e9ea0c046828e9c8dbfedc266101bd78975b79f587da363163bd32f94"} Oct 30 20:57:10.294487 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.294468 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp" event={"ID":"1e3d6326-166f-47f0-888e-f7a5e8464aa1","Type":"ContainerDied","Data":"101d045a05929d808bed90581fff265cd6da592ece6960ba5313b147be1ecf70"} Oct 30 20:57:10.303208 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.303178 5755 scope.go:122] "RemoveContainer" containerID="9cda15cfe7c97261babd672c204d36facc1f1b977bc65661a298e2eb33529ca8" Oct 30 20:57:10.303470 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:57:10.303451 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cda15cfe7c97261babd672c204d36facc1f1b977bc65661a298e2eb33529ca8\": container with ID starting with 9cda15cfe7c97261babd672c204d36facc1f1b977bc65661a298e2eb33529ca8 not found: ID does not exist" containerID="9cda15cfe7c97261babd672c204d36facc1f1b977bc65661a298e2eb33529ca8" Oct 30 20:57:10.303555 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.303477 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cda15cfe7c97261babd672c204d36facc1f1b977bc65661a298e2eb33529ca8"} err="failed to get container status \"9cda15cfe7c97261babd672c204d36facc1f1b977bc65661a298e2eb33529ca8\": rpc error: code = NotFound desc = could not find container \"9cda15cfe7c97261babd672c204d36facc1f1b977bc65661a298e2eb33529ca8\": container with ID starting with 9cda15cfe7c97261babd672c204d36facc1f1b977bc65661a298e2eb33529ca8 not found: ID does not exist" Oct 30 20:57:10.303555 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.303493 5755 scope.go:122] "RemoveContainer" containerID="696e390e9ea0c046828e9c8dbfedc266101bd78975b79f587da363163bd32f94" Oct 30 20:57:10.312923 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.312906 5755 scope.go:122] "RemoveContainer" containerID="696e390e9ea0c046828e9c8dbfedc266101bd78975b79f587da363163bd32f94" Oct 30 20:57:10.313183 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:57:10.313162 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"696e390e9ea0c046828e9c8dbfedc266101bd78975b79f587da363163bd32f94\": container with ID starting with 696e390e9ea0c046828e9c8dbfedc266101bd78975b79f587da363163bd32f94 not found: ID does not exist" containerID="696e390e9ea0c046828e9c8dbfedc266101bd78975b79f587da363163bd32f94" Oct 30 20:57:10.313263 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.313187 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696e390e9ea0c046828e9c8dbfedc266101bd78975b79f587da363163bd32f94"} err="failed to get container status \"696e390e9ea0c046828e9c8dbfedc266101bd78975b79f587da363163bd32f94\": rpc error: code = NotFound desc = could not find container \"696e390e9ea0c046828e9c8dbfedc266101bd78975b79f587da363163bd32f94\": container with ID starting with 696e390e9ea0c046828e9c8dbfedc266101bd78975b79f587da363163bd32f94 not found: ID does not exist" Oct 30 20:57:10.561626 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.561544 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b"] Oct 30 20:57:10.610967 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.610927 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-hhz9b"] Oct 30 20:57:10.681679 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.681644 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp"] Oct 30 20:57:10.688639 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:10.688606 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-vknjp"] Oct 30 20:57:11.213555 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:11.213523 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3d6326-166f-47f0-888e-f7a5e8464aa1" path="/var/lib/kubelet/pods/1e3d6326-166f-47f0-888e-f7a5e8464aa1/volumes" Oct 30 20:57:11.213997 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:11.213984 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" path="/var/lib/kubelet/pods/80aa1317-6c2c-48b4-a5f5-bbfa7928f08a/volumes" Oct 30 20:57:11.255578 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:57:11.255544 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:57:27.635794 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:27.635673 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd"] Oct 30 20:57:27.684822 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:27.684778 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0702ff-b8zpd"] Oct 30 20:57:29.213069 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:29.213035 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c57fe592-756b-47a1-ac10-09836af845f4" path="/var/lib/kubelet/pods/c57fe592-756b-47a1-ac10-09836af845f4/volumes" Oct 30 20:57:35.151941 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:35.151906 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v"] Oct 30 20:57:35.152366 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:35.152218 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" podUID="ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" containerName="istio-proxy" containerID="cri-o://fdeb707e66f2412af9f6b08aa23b02a9377de6e5c81b415bfc8e94d0b105a02f" gracePeriod=30 Oct 30 20:57:35.987606 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:35.987571 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz"] Oct 30 20:57:35.988083 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:35.988071 5755 cpu_manager.go:414] "containerMap: removing container" podUID="70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3" containerName="revoke-keys" Oct 30 20:57:35.988134 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:35.988085 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3" containerName="revoke-keys" Oct 30 20:57:35.988134 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:35.988107 5755 cpu_manager.go:414] "containerMap: removing container" podUID="b0906f40-8902-4c30-b0ea-7576edea2f8c" containerName="revoke-keys" Oct 30 20:57:35.988134 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:35.988112 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0906f40-8902-4c30-b0ea-7576edea2f8c" containerName="revoke-keys" Oct 30 20:57:35.988134 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:35.988124 5755 cpu_manager.go:414] "containerMap: removing container" podUID="80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" containerName="istio-proxy" Oct 30 20:57:35.988134 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:35.988129 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" containerName="istio-proxy" Oct 30 20:57:35.988288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:35.988143 5755 cpu_manager.go:414] "containerMap: removing container" podUID="1e3d6326-166f-47f0-888e-f7a5e8464aa1" containerName="istio-proxy" Oct 30 20:57:35.988288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:35.988147 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3d6326-166f-47f0-888e-f7a5e8464aa1" containerName="istio-proxy" Oct 30 20:57:35.988288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:35.988234 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0906f40-8902-4c30-b0ea-7576edea2f8c" containerName="revoke-keys" Oct 30 20:57:35.988288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:35.988243 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="80aa1317-6c2c-48b4-a5f5-bbfa7928f08a" containerName="istio-proxy" Oct 30 20:57:35.988288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:35.988252 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3d6326-166f-47f0-888e-f7a5e8464aa1" containerName="istio-proxy" Oct 30 20:57:35.988288 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:35.988262 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3" containerName="revoke-keys" Oct 30 20:57:35.991803 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:35.991787 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" Oct 30 20:57:36.048221 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:36.048184 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz"] Oct 30 20:57:36.094010 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:36.093979 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:57:36.094204 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:36.094171 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:57:36.114263 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:36.114221 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz\" (UID: \"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" Oct 30 20:57:36.114263 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:36.114267 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz\" (UID: \"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" Oct 30 20:57:36.114503 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:36.114286 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp97m\" (UniqueName: \"kubernetes.io/projected/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-kube-api-access-kp97m\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz\" (UID: \"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" Oct 30 20:57:36.215497 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:36.215435 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz\" (UID: \"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" Oct 30 20:57:36.215910 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:36.215511 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp97m\" (UniqueName: \"kubernetes.io/projected/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-kube-api-access-kp97m\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz\" (UID: \"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" Oct 30 20:57:36.215910 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:36.215770 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz\" (UID: \"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" Oct 30 20:57:36.216321 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:36.216300 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz\" (UID: \"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" Oct 30 20:57:36.217864 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:36.217846 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz\" (UID: \"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" Oct 30 20:57:36.274305 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:36.274238 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp97m\" (UniqueName: \"kubernetes.io/projected/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-kube-api-access-kp97m\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz\" (UID: \"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" Oct 30 20:57:36.305602 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:36.305572 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" Oct 30 20:57:36.689560 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:36.689524 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz"] Oct 30 20:57:36.690516 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:57:36.690488 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18683c1f_ed8e_4a35_be3b_7ef8b3795e6e.slice/crio-e00b42321dad44ea19d1fc93d1c275c33059960b1927d4549e86b953cb2f4b21 WatchSource:0}: Error finding container e00b42321dad44ea19d1fc93d1c275c33059960b1927d4549e86b953cb2f4b21: Status 404 returned error can't find the container with id e00b42321dad44ea19d1fc93d1c275c33059960b1927d4549e86b953cb2f4b21 Oct 30 20:57:37.544599 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:37.544559 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" event={"ID":"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e","Type":"ContainerStarted","Data":"c9a2c2074bac0c18e01dbc7777f3eadded32065341d77396d5a4ae4ad3d6c128"} Oct 30 20:57:37.544996 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:37.544640 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" event={"ID":"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e","Type":"ContainerStarted","Data":"e00b42321dad44ea19d1fc93d1c275c33059960b1927d4549e86b953cb2f4b21"} Oct 30 20:57:37.661052 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:37.660995 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" podStartSLOduration=2.660981602 podStartE2EDuration="2.660981602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:37.660093085 +0000 UTC m=+1820.586650002" watchObservedRunningTime="2026-10-30 20:57:37.660981602 +0000 UTC m=+1820.587538566" Oct 30 20:57:38.554972 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:38.554937 5755 generic.go:356] "Generic (PLEG): container finished" podID="18683c1f-ed8e-4a35-be3b-7ef8b3795e6e" containerID="c9a2c2074bac0c18e01dbc7777f3eadded32065341d77396d5a4ae4ad3d6c128" exitCode=0 Oct 30 20:57:38.555450 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:38.555024 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" event={"ID":"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e","Type":"ContainerDied","Data":"c9a2c2074bac0c18e01dbc7777f3eadded32065341d77396d5a4ae4ad3d6c128"} Oct 30 20:57:39.708175 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:39.708145 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" Oct 30 20:57:39.866736 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:39.866640 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-kube-api-access-kp97m\" (UniqueName: \"kubernetes.io/projected/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-kube-api-access-kp97m\") pod \"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e\" (UID: \"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e\") " Oct 30 20:57:39.866913 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:39.866853 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-maas-api-openshift-ca\") pod \"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e\" (UID: \"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e\") " Oct 30 20:57:39.866998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:39.866981 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-maas-api-xks-ca\") pod \"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e\" (UID: \"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e\") " Oct 30 20:57:39.867283 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:39.867259 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-maas-api-openshift-ca" pod "18683c1f-ed8e-4a35-be3b-7ef8b3795e6e" (UID: "18683c1f-ed8e-4a35-be3b-7ef8b3795e6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:39.867489 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:39.867473 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:39.869059 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:39.869036 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-kube-api-access-kp97m" pod "18683c1f-ed8e-4a35-be3b-7ef8b3795e6e" (UID: "18683c1f-ed8e-4a35-be3b-7ef8b3795e6e"). InnerVolumeSpecName "kube-api-access-kp97m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:39.869155 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:39.869066 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-maas-api-xks-ca" pod "18683c1f-ed8e-4a35-be3b-7ef8b3795e6e" (UID: "18683c1f-ed8e-4a35-be3b-7ef8b3795e6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:39.967458 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:39.967421 5755 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:57:39.967645 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:39.967480 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" podUID="ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:57:39.968416 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:39.968355 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp97m\" (UniqueName: \"kubernetes.io/projected/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-kube-api-access-kp97m\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:39.968416 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:39.968416 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:40.403381 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.403351 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:57:40.473336 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.473305 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-token\" (UniqueName: \"kubernetes.io/projected/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-token\") pod \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " Oct 30 20:57:40.473579 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.473414 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istiod-ca-cert\") pod \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " Oct 30 20:57:40.473579 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.473489 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-kube-api-access-9t2tv\" (UniqueName: \"kubernetes.io/projected/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-kube-api-access-9t2tv\") pod \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " Oct 30 20:57:40.473579 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.473540 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-workload-certs\") pod \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " Oct 30 20:57:40.473762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.473598 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-podinfo\") pod \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " Oct 30 20:57:40.473762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.473644 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-data\") pod \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " Oct 30 20:57:40.473762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.473692 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-envoy\") pod \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " Oct 30 20:57:40.473924 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.473765 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-workload-socket\") pod \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " Oct 30 20:57:40.473924 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.473815 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-credential-socket\") pod \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\" (UID: \"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc\") " Oct 30 20:57:40.473924 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.473875 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-workload-certs" pod "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" (UID: "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:40.474087 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.473958 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-data" pod "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" (UID: "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:40.474087 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.473970 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istiod-ca-cert" pod "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" (UID: "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:40.474200 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.474167 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-credential-socket" pod "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" (UID: "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:40.474349 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.474325 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-workload-socket" pod "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" (UID: "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:40.474470 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.474454 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:40.474536 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.474476 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:40.474536 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.474491 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:40.474536 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.474506 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:40.475831 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.475810 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-token" pod "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" (UID: "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:40.475956 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.475876 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-podinfo" pod "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" (UID: "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:40.476103 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.476084 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-envoy" pod "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" (UID: "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:40.476424 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.476402 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-kube-api-access-9t2tv" pod "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" (UID: "ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc"). InnerVolumeSpecName "kube-api-access-9t2tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:40.569962 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.569921 5755 generic.go:356] "Generic (PLEG): container finished" podID="ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" containerID="fdeb707e66f2412af9f6b08aa23b02a9377de6e5c81b415bfc8e94d0b105a02f" exitCode=0 Oct 30 20:57:40.570133 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.569973 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" Oct 30 20:57:40.570133 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.570006 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" event={"ID":"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc","Type":"ContainerDied","Data":"fdeb707e66f2412af9f6b08aa23b02a9377de6e5c81b415bfc8e94d0b105a02f"} Oct 30 20:57:40.570133 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.570065 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v" event={"ID":"ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc","Type":"ContainerDied","Data":"d5aa16d1e690096144175964a81844d428b979a608692238adf8577ae6ea91d3"} Oct 30 20:57:40.570133 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.570095 5755 scope.go:122] "RemoveContainer" containerID="fdeb707e66f2412af9f6b08aa23b02a9377de6e5c81b415bfc8e94d0b105a02f" Oct 30 20:57:40.571968 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.571907 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" Oct 30 20:57:40.571968 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.571931 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz" event={"ID":"18683c1f-ed8e-4a35-be3b-7ef8b3795e6e","Type":"ContainerDied","Data":"e00b42321dad44ea19d1fc93d1c275c33059960b1927d4549e86b953cb2f4b21"} Oct 30 20:57:40.572141 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.571975 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e00b42321dad44ea19d1fc93d1c275c33059960b1927d4549e86b953cb2f4b21" Oct 30 20:57:40.576011 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.575984 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:40.576139 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.576015 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:40.576139 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.576030 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t2tv\" (UniqueName: \"kubernetes.io/projected/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-kube-api-access-9t2tv\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:40.576139 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.576043 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:40.576139 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.576056 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:40.583727 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.583703 5755 scope.go:122] "RemoveContainer" containerID="fdeb707e66f2412af9f6b08aa23b02a9377de6e5c81b415bfc8e94d0b105a02f" Oct 30 20:57:40.584086 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:57:40.584060 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdeb707e66f2412af9f6b08aa23b02a9377de6e5c81b415bfc8e94d0b105a02f\": container with ID starting with fdeb707e66f2412af9f6b08aa23b02a9377de6e5c81b415bfc8e94d0b105a02f not found: ID does not exist" containerID="fdeb707e66f2412af9f6b08aa23b02a9377de6e5c81b415bfc8e94d0b105a02f" Oct 30 20:57:40.584182 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.584100 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdeb707e66f2412af9f6b08aa23b02a9377de6e5c81b415bfc8e94d0b105a02f"} err="failed to get container status \"fdeb707e66f2412af9f6b08aa23b02a9377de6e5c81b415bfc8e94d0b105a02f\": rpc error: code = NotFound desc = could not find container \"fdeb707e66f2412af9f6b08aa23b02a9377de6e5c81b415bfc8e94d0b105a02f\": container with ID starting with fdeb707e66f2412af9f6b08aa23b02a9377de6e5c81b415bfc8e94d0b105a02f not found: ID does not exist" Oct 30 20:57:40.738353 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.738267 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v"] Oct 30 20:57:40.745671 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:40.745644 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-s9r6v"] Oct 30 20:57:41.219989 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:41.219951 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" path="/var/lib/kubelet/pods/ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc/volumes" Oct 30 20:57:41.257899 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:57:41.257860 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:57:42.051231 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:42.051191 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc"] Oct 30 20:57:42.054910 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:42.054883 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c05a66-tqnxc"] Oct 30 20:57:43.212770 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:43.212728 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24db4576-f3e1-475a-aa4b-ea0b7d66fbf7" path="/var/lib/kubelet/pods/24db4576-f3e1-475a-aa4b-ea0b7d66fbf7/volumes" Oct 30 20:57:44.894395 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:44.894347 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx"] Oct 30 20:57:44.944020 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:44.943991 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c83a089-9fflx"] Oct 30 20:57:45.212690 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:45.212612 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0906f40-8902-4c30-b0ea-7576edea2f8c" path="/var/lib/kubelet/pods/b0906f40-8902-4c30-b0ea-7576edea2f8c/volumes" Oct 30 20:57:53.371079 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.371040 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk"] Oct 30 20:57:53.371709 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.371694 5755 cpu_manager.go:414] "containerMap: removing container" podUID="18683c1f-ed8e-4a35-be3b-7ef8b3795e6e" containerName="revoke-keys" Oct 30 20:57:53.371769 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.371711 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18683c1f-ed8e-4a35-be3b-7ef8b3795e6e" containerName="revoke-keys" Oct 30 20:57:53.371769 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.371747 5755 cpu_manager.go:414] "containerMap: removing container" podUID="ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" containerName="istio-proxy" Oct 30 20:57:53.371769 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.371752 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" containerName="istio-proxy" Oct 30 20:57:53.371875 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.371828 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="18683c1f-ed8e-4a35-be3b-7ef8b3795e6e" containerName="revoke-keys" Oct 30 20:57:53.371875 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.371843 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5cbd75-bc8a-4dee-b7ac-54fbdd2b81fc" containerName="istio-proxy" Oct 30 20:57:53.376645 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.376630 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" Oct 30 20:57:53.433022 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.432994 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk"] Oct 30 20:57:53.478341 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.478308 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:57:53.478341 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.478321 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:57:53.495338 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.495309 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z675\" (UniqueName: \"kubernetes.io/projected/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-kube-api-access-6z675\") pod \"maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk\" (UID: \"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" Oct 30 20:57:53.495499 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.495401 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk\" (UID: \"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" Oct 30 20:57:53.495499 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.495438 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk\" (UID: \"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" Oct 30 20:57:53.596814 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.596779 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z675\" (UniqueName: \"kubernetes.io/projected/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-kube-api-access-6z675\") pod \"maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk\" (UID: \"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" Oct 30 20:57:53.596993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.596862 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk\" (UID: \"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" Oct 30 20:57:53.596993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.596902 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk\" (UID: \"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" Oct 30 20:57:53.597691 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.597672 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk\" (UID: \"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" Oct 30 20:57:53.599495 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.599479 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk\" (UID: \"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" Oct 30 20:57:53.656704 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.656633 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z675\" (UniqueName: \"kubernetes.io/projected/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-kube-api-access-6z675\") pod \"maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk\" (UID: \"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" Oct 30 20:57:53.690275 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:53.690246 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" Oct 30 20:57:54.070580 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:54.070555 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk"] Oct 30 20:57:54.071695 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:57:54.071666 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6b7bdc_b19c_4370_8d36_1d90c4e8e63f.slice/crio-0def172c99ac6ca65ad776235135bdf92f87975f7099c23bebbb22ae65cf68d3 WatchSource:0}: Error finding container 0def172c99ac6ca65ad776235135bdf92f87975f7099c23bebbb22ae65cf68d3: Status 404 returned error can't find the container with id 0def172c99ac6ca65ad776235135bdf92f87975f7099c23bebbb22ae65cf68d3 Oct 30 20:57:54.667631 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:54.667551 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" event={"ID":"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f","Type":"ContainerStarted","Data":"ea03a1cdf1612a04227d871bc6b1b205451595e1e3befa11e33d92ceb8f00904"} Oct 30 20:57:54.667631 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:54.667611 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" event={"ID":"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f","Type":"ContainerStarted","Data":"0def172c99ac6ca65ad776235135bdf92f87975f7099c23bebbb22ae65cf68d3"} Oct 30 20:57:54.785995 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:54.785946 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" podStartSLOduration=1.7859312790000001 podStartE2EDuration="1.785931279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:54.783895075 +0000 UTC m=+1837.710452017" watchObservedRunningTime="2026-10-30 20:57:54.785931279 +0000 UTC m=+1837.712488194" Oct 30 20:57:55.675857 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:55.675825 5755 generic.go:356] "Generic (PLEG): container finished" podID="dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f" containerID="ea03a1cdf1612a04227d871bc6b1b205451595e1e3befa11e33d92ceb8f00904" exitCode=0 Oct 30 20:57:55.676283 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:55.675909 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" event={"ID":"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f","Type":"ContainerDied","Data":"ea03a1cdf1612a04227d871bc6b1b205451595e1e3befa11e33d92ceb8f00904"} Oct 30 20:57:56.835964 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:56.835933 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" Oct 30 20:57:56.948141 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:56.948061 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-maas-api-openshift-ca\") pod \"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f\" (UID: \"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f\") " Oct 30 20:57:56.948141 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:56.948096 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-maas-api-xks-ca\") pod \"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f\" (UID: \"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f\") " Oct 30 20:57:56.948415 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:56.948194 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-kube-api-access-6z675\" (UniqueName: \"kubernetes.io/projected/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-kube-api-access-6z675\") pod \"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f\" (UID: \"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f\") " Oct 30 20:57:56.948536 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:56.948512 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-maas-api-openshift-ca" pod "dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f" (UID: "dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:56.950514 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:56.950485 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-maas-api-xks-ca" pod "dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f" (UID: "dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:56.950622 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:56.950523 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-kube-api-access-6z675" pod "dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f" (UID: "dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f"). InnerVolumeSpecName "kube-api-access-6z675". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:57.049530 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:57.049497 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:57.049530 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:57.049527 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:57.049530 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:57.049537 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z675\" (UniqueName: \"kubernetes.io/projected/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f-kube-api-access-6z675\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:57.691054 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:57.690968 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" event={"ID":"dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f","Type":"ContainerDied","Data":"0def172c99ac6ca65ad776235135bdf92f87975f7099c23bebbb22ae65cf68d3"} Oct 30 20:57:57.691054 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:57.690999 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk" Oct 30 20:57:57.691054 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:57.691036 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0def172c99ac6ca65ad776235135bdf92f87975f7099c23bebbb22ae65cf68d3" Oct 30 20:57:57.798737 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:57.798704 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59"] Oct 30 20:57:57.805753 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:57.805727 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7"] Oct 30 20:57:57.840434 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:57:57.840379 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq"] Oct 30 20:58:05.044529 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:05.044491 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn"] Oct 30 20:58:07.286800 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:07.286768 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9"] Oct 30 20:58:07.336974 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:07.336945 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aaf669-4nvf9"] Oct 30 20:58:09.212504 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:09.212470 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3" path="/var/lib/kubelet/pods/70a7ae95-219f-4ed4-a8df-2fd2ef61d9f3/volumes" Oct 30 20:58:10.045335 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:10.045287 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" podUID="312ff01e-18e1-4d99-9dfd-67df3b8e640f" containerName="payload-processing" containerID="cri-o://9a3703de5f02d5004b7709efc5cf47b6fd047f268008d977b1225b1229b02a39" gracePeriod=25 Oct 30 20:58:11.255977 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:58:11.255938 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:58:12.821065 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:12.821026 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" containerID="cri-o://dcd8ceaf26047b1879641285c433580f16f53e2347eb0b1805fa50c85966986c" gracePeriod=45 Oct 30 20:58:12.826352 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:12.826301 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" containerID="cri-o://c7a0c66c46121bdf79f94f1f7b6d82499d86ee878366b0f51b7f19c482aa6328" gracePeriod=45 Oct 30 20:58:12.860003 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:12.859972 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" containerID="cri-o://949e13b714e0079aa32fd5e3ab1a9bd89db7d2453b132a8988022f896bb84af5" gracePeriod=45 Oct 30 20:58:13.169248 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.169222 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:58:13.172634 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.172613 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:58:13.182591 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.182565 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:58:13.232018 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.231986 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" podUID="312ff01e-18e1-4d99-9dfd-67df3b8e640f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:58:13.330347 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.330254 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-model-cache\") pod \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " Oct 30 20:58:13.330347 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.330324 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a305512d-f7b3-4b29-b72e-22115f7c8b0c-tls-certs\" (UniqueName: \"kubernetes.io/secret/a305512d-f7b3-4b29-b72e-22115f7c8b0c-tls-certs\") pod \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " Oct 30 20:58:13.330613 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.330356 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-home\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-home\") pod \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " Oct 30 20:58:13.330613 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.330472 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-model-cache\") pod \"197b515b-ee40-4966-aa7f-f20e16001039\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " Oct 30 20:58:13.330613 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.330499 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-dshm\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-dshm\") pod \"197b515b-ee40-4966-aa7f-f20e16001039\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " Oct 30 20:58:13.330613 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.330528 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197b515b-ee40-4966-aa7f-f20e16001039-kube-api-access-qzwvq\" (UniqueName: \"kubernetes.io/projected/197b515b-ee40-4966-aa7f-f20e16001039-kube-api-access-qzwvq\") pod \"197b515b-ee40-4966-aa7f-f20e16001039\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " Oct 30 20:58:13.330613 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.330569 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-tmp-dir\") pod \"197b515b-ee40-4966-aa7f-f20e16001039\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " Oct 30 20:58:13.330613 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.330585 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-model-cache" pod "a305512d-f7b3-4b29-b72e-22115f7c8b0c" (UID: "a305512d-f7b3-4b29-b72e-22115f7c8b0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:13.330613 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.330611 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-dshm\") pod \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " Oct 30 20:58:13.330978 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.330673 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-home\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-home\") pod \"68b1092f-af0f-477c-81c0-cc3083b7f564\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " Oct 30 20:58:13.330978 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.330700 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-tmp-dir\") pod \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " Oct 30 20:58:13.330978 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.330757 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68b1092f-af0f-477c-81c0-cc3083b7f564-tls-certs\" (UniqueName: \"kubernetes.io/secret/68b1092f-af0f-477c-81c0-cc3083b7f564-tls-certs\") pod \"68b1092f-af0f-477c-81c0-cc3083b7f564\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " Oct 30 20:58:13.330978 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.330789 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-model-cache" pod "197b515b-ee40-4966-aa7f-f20e16001039" (UID: "197b515b-ee40-4966-aa7f-f20e16001039"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:13.330978 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.330793 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b1092f-af0f-477c-81c0-cc3083b7f564-kube-api-access-j9q4r\" (UniqueName: \"kubernetes.io/projected/68b1092f-af0f-477c-81c0-cc3083b7f564-kube-api-access-j9q4r\") pod \"68b1092f-af0f-477c-81c0-cc3083b7f564\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " Oct 30 20:58:13.331263 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.331028 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-home" pod "a305512d-f7b3-4b29-b72e-22115f7c8b0c" (UID: "a305512d-f7b3-4b29-b72e-22115f7c8b0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:13.331263 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.331214 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-tmp-dir" pod "197b515b-ee40-4966-aa7f-f20e16001039" (UID: "197b515b-ee40-4966-aa7f-f20e16001039"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:13.331467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.331444 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-home" pod "68b1092f-af0f-477c-81c0-cc3083b7f564" (UID: "68b1092f-af0f-477c-81c0-cc3083b7f564"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:13.331579 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.331497 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a305512d-f7b3-4b29-b72e-22115f7c8b0c-kube-api-access-vjktn\" (UniqueName: \"kubernetes.io/projected/a305512d-f7b3-4b29-b72e-22115f7c8b0c-kube-api-access-vjktn\") pod \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\" (UID: \"a305512d-f7b3-4b29-b72e-22115f7c8b0c\") " Oct 30 20:58:13.331579 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.331556 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-home\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-home\") pod \"197b515b-ee40-4966-aa7f-f20e16001039\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " Oct 30 20:58:13.331704 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.331641 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-dshm\") pod \"68b1092f-af0f-477c-81c0-cc3083b7f564\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " Oct 30 20:58:13.331763 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.331704 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-model-cache\") pod \"68b1092f-af0f-477c-81c0-cc3083b7f564\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " Oct 30 20:58:13.331763 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.331744 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-tmp-dir\") pod \"68b1092f-af0f-477c-81c0-cc3083b7f564\" (UID: \"68b1092f-af0f-477c-81c0-cc3083b7f564\") " Oct 30 20:58:13.331873 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.331797 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/197b515b-ee40-4966-aa7f-f20e16001039-tls-certs\" (UniqueName: \"kubernetes.io/secret/197b515b-ee40-4966-aa7f-f20e16001039-tls-certs\") pod \"197b515b-ee40-4966-aa7f-f20e16001039\" (UID: \"197b515b-ee40-4966-aa7f-f20e16001039\") " Oct 30 20:58:13.331927 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.331887 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-tmp-dir" pod "a305512d-f7b3-4b29-b72e-22115f7c8b0c" (UID: "a305512d-f7b3-4b29-b72e-22115f7c8b0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:13.332556 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.332526 5755 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-tmp-dir\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.332556 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.332557 5755 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-home\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.332739 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.332571 5755 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-tmp-dir\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.332739 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.332584 5755 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-model-cache\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.332739 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.332597 5755 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-home\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.332739 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.332610 5755 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-model-cache\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.334030 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.334003 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b1092f-af0f-477c-81c0-cc3083b7f564-kube-api-access-j9q4r" pod "68b1092f-af0f-477c-81c0-cc3083b7f564" (UID: "68b1092f-af0f-477c-81c0-cc3083b7f564"). InnerVolumeSpecName "kube-api-access-j9q4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:13.334316 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.334292 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-model-cache" pod "68b1092f-af0f-477c-81c0-cc3083b7f564" (UID: "68b1092f-af0f-477c-81c0-cc3083b7f564"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:13.334670 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.334624 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-dshm" pod "197b515b-ee40-4966-aa7f-f20e16001039" (UID: "197b515b-ee40-4966-aa7f-f20e16001039"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:13.334670 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.334646 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-tmp-dir" pod "68b1092f-af0f-477c-81c0-cc3083b7f564" (UID: "68b1092f-af0f-477c-81c0-cc3083b7f564"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:13.334894 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.334873 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-home" pod "197b515b-ee40-4966-aa7f-f20e16001039" (UID: "197b515b-ee40-4966-aa7f-f20e16001039"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:13.335105 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.335076 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/197b515b-ee40-4966-aa7f-f20e16001039-tls-certs" pod "197b515b-ee40-4966-aa7f-f20e16001039" (UID: "197b515b-ee40-4966-aa7f-f20e16001039"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:13.335761 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.335723 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-dshm" pod "68b1092f-af0f-477c-81c0-cc3083b7f564" (UID: "68b1092f-af0f-477c-81c0-cc3083b7f564"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:13.336675 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.336478 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197b515b-ee40-4966-aa7f-f20e16001039-kube-api-access-qzwvq" pod "197b515b-ee40-4966-aa7f-f20e16001039" (UID: "197b515b-ee40-4966-aa7f-f20e16001039"). InnerVolumeSpecName "kube-api-access-qzwvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:13.337155 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.337128 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b1092f-af0f-477c-81c0-cc3083b7f564-tls-certs" pod "68b1092f-af0f-477c-81c0-cc3083b7f564" (UID: "68b1092f-af0f-477c-81c0-cc3083b7f564"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:13.337461 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.337336 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a305512d-f7b3-4b29-b72e-22115f7c8b0c-kube-api-access-vjktn" pod "a305512d-f7b3-4b29-b72e-22115f7c8b0c" (UID: "a305512d-f7b3-4b29-b72e-22115f7c8b0c"). InnerVolumeSpecName "kube-api-access-vjktn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:13.338166 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.337828 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-dshm" pod "a305512d-f7b3-4b29-b72e-22115f7c8b0c" (UID: "a305512d-f7b3-4b29-b72e-22115f7c8b0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:13.338166 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.338140 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a305512d-f7b3-4b29-b72e-22115f7c8b0c-tls-certs" pod "a305512d-f7b3-4b29-b72e-22115f7c8b0c" (UID: "a305512d-f7b3-4b29-b72e-22115f7c8b0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:13.433816 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.433782 5755 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-dshm\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.433816 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.433816 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzwvq\" (UniqueName: \"kubernetes.io/projected/197b515b-ee40-4966-aa7f-f20e16001039-kube-api-access-qzwvq\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.434020 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.433833 5755 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a305512d-f7b3-4b29-b72e-22115f7c8b0c-dshm\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.434020 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.433848 5755 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b1092f-af0f-477c-81c0-cc3083b7f564-tls-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.434020 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.433862 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9q4r\" (UniqueName: \"kubernetes.io/projected/68b1092f-af0f-477c-81c0-cc3083b7f564-kube-api-access-j9q4r\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.434020 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.433876 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjktn\" (UniqueName: \"kubernetes.io/projected/a305512d-f7b3-4b29-b72e-22115f7c8b0c-kube-api-access-vjktn\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.434020 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.433891 5755 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197b515b-ee40-4966-aa7f-f20e16001039-home\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.434020 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.433904 5755 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-dshm\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.434020 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.433917 5755 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-model-cache\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.434020 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.433930 5755 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b1092f-af0f-477c-81c0-cc3083b7f564-tmp-dir\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.434020 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.433944 5755 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197b515b-ee40-4966-aa7f-f20e16001039-tls-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.434020 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.433956 5755 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a305512d-f7b3-4b29-b72e-22115f7c8b0c-tls-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:13.832453 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.832413 5755 generic.go:356] "Generic (PLEG): container finished" podID="197b515b-ee40-4966-aa7f-f20e16001039" containerID="949e13b714e0079aa32fd5e3ab1a9bd89db7d2453b132a8988022f896bb84af5" exitCode=0 Oct 30 20:58:13.832925 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.832471 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" Oct 30 20:58:13.832925 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.832488 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" event={"ID":"197b515b-ee40-4966-aa7f-f20e16001039","Type":"ContainerDied","Data":"949e13b714e0079aa32fd5e3ab1a9bd89db7d2453b132a8988022f896bb84af5"} Oct 30 20:58:13.832925 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.832536 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq" event={"ID":"197b515b-ee40-4966-aa7f-f20e16001039","Type":"ContainerDied","Data":"b2ae8c98bac79c0c08a530578ea0be9b6d852446e6c137982311e10439b8e76e"} Oct 30 20:58:13.832925 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.832554 5755 scope.go:122] "RemoveContainer" containerID="949e13b714e0079aa32fd5e3ab1a9bd89db7d2453b132a8988022f896bb84af5" Oct 30 20:58:13.834345 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.834319 5755 generic.go:356] "Generic (PLEG): container finished" podID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerID="dcd8ceaf26047b1879641285c433580f16f53e2347eb0b1805fa50c85966986c" exitCode=0 Oct 30 20:58:13.834490 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.834379 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" event={"ID":"a305512d-f7b3-4b29-b72e-22115f7c8b0c","Type":"ContainerDied","Data":"dcd8ceaf26047b1879641285c433580f16f53e2347eb0b1805fa50c85966986c"} Oct 30 20:58:13.834490 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.834431 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" event={"ID":"a305512d-f7b3-4b29-b72e-22115f7c8b0c","Type":"ContainerDied","Data":"7f7a82ab5d2d7aec087c756561bd8dc54a679e1f80760a4549fc7cb14fc1e596"} Oct 30 20:58:13.834490 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.834430 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59" Oct 30 20:58:13.836207 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.836183 5755 generic.go:356] "Generic (PLEG): container finished" podID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerID="c7a0c66c46121bdf79f94f1f7b6d82499d86ee878366b0f51b7f19c482aa6328" exitCode=0 Oct 30 20:58:13.836304 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.836237 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" event={"ID":"68b1092f-af0f-477c-81c0-cc3083b7f564","Type":"ContainerDied","Data":"c7a0c66c46121bdf79f94f1f7b6d82499d86ee878366b0f51b7f19c482aa6328"} Oct 30 20:58:13.836304 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.836265 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" event={"ID":"68b1092f-af0f-477c-81c0-cc3083b7f564","Type":"ContainerDied","Data":"19bd8766b1a2951bec3e8f44988653b5ada7dae665eca6f9be00ce4365ea8241"} Oct 30 20:58:13.836459 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.836329 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7" Oct 30 20:58:13.860121 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.860101 5755 scope.go:122] "RemoveContainer" containerID="949e13b714e0079aa32fd5e3ab1a9bd89db7d2453b132a8988022f896bb84af5" Oct 30 20:58:13.860675 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:58:13.860650 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949e13b714e0079aa32fd5e3ab1a9bd89db7d2453b132a8988022f896bb84af5\": container with ID starting with 949e13b714e0079aa32fd5e3ab1a9bd89db7d2453b132a8988022f896bb84af5 not found: ID does not exist" containerID="949e13b714e0079aa32fd5e3ab1a9bd89db7d2453b132a8988022f896bb84af5" Oct 30 20:58:13.860793 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.860685 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949e13b714e0079aa32fd5e3ab1a9bd89db7d2453b132a8988022f896bb84af5"} err="failed to get container status \"949e13b714e0079aa32fd5e3ab1a9bd89db7d2453b132a8988022f896bb84af5\": rpc error: code = NotFound desc = could not find container \"949e13b714e0079aa32fd5e3ab1a9bd89db7d2453b132a8988022f896bb84af5\": container with ID starting with 949e13b714e0079aa32fd5e3ab1a9bd89db7d2453b132a8988022f896bb84af5 not found: ID does not exist" Oct 30 20:58:13.860793 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.860708 5755 scope.go:122] "RemoveContainer" containerID="dcd8ceaf26047b1879641285c433580f16f53e2347eb0b1805fa50c85966986c" Oct 30 20:58:13.870683 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.870667 5755 scope.go:122] "RemoveContainer" containerID="dcd8ceaf26047b1879641285c433580f16f53e2347eb0b1805fa50c85966986c" Oct 30 20:58:13.870970 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:58:13.870949 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd8ceaf26047b1879641285c433580f16f53e2347eb0b1805fa50c85966986c\": container with ID starting with dcd8ceaf26047b1879641285c433580f16f53e2347eb0b1805fa50c85966986c not found: ID does not exist" containerID="dcd8ceaf26047b1879641285c433580f16f53e2347eb0b1805fa50c85966986c" Oct 30 20:58:13.871031 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.870977 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd8ceaf26047b1879641285c433580f16f53e2347eb0b1805fa50c85966986c"} err="failed to get container status \"dcd8ceaf26047b1879641285c433580f16f53e2347eb0b1805fa50c85966986c\": rpc error: code = NotFound desc = could not find container \"dcd8ceaf26047b1879641285c433580f16f53e2347eb0b1805fa50c85966986c\": container with ID starting with dcd8ceaf26047b1879641285c433580f16f53e2347eb0b1805fa50c85966986c not found: ID does not exist" Oct 30 20:58:13.871031 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.870992 5755 scope.go:122] "RemoveContainer" containerID="c7a0c66c46121bdf79f94f1f7b6d82499d86ee878366b0f51b7f19c482aa6328" Oct 30 20:58:13.879475 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.879459 5755 scope.go:122] "RemoveContainer" containerID="c7a0c66c46121bdf79f94f1f7b6d82499d86ee878366b0f51b7f19c482aa6328" Oct 30 20:58:13.879738 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:58:13.879719 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a0c66c46121bdf79f94f1f7b6d82499d86ee878366b0f51b7f19c482aa6328\": container with ID starting with c7a0c66c46121bdf79f94f1f7b6d82499d86ee878366b0f51b7f19c482aa6328 not found: ID does not exist" containerID="c7a0c66c46121bdf79f94f1f7b6d82499d86ee878366b0f51b7f19c482aa6328" Oct 30 20:58:13.879793 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:13.879745 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a0c66c46121bdf79f94f1f7b6d82499d86ee878366b0f51b7f19c482aa6328"} err="failed to get container status \"c7a0c66c46121bdf79f94f1f7b6d82499d86ee878366b0f51b7f19c482aa6328\": rpc error: code = NotFound desc = could not find container \"c7a0c66c46121bdf79f94f1f7b6d82499d86ee878366b0f51b7f19c482aa6328\": container with ID starting with c7a0c66c46121bdf79f94f1f7b6d82499d86ee878366b0f51b7f19c482aa6328 not found: ID does not exist" Oct 30 20:58:14.000521 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:14.000486 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq"] Oct 30 20:58:14.050982 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:14.050954 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/simulator-w3-f6ebf5-kserve-59ffc9879b-pnfhq"] Oct 30 20:58:14.116154 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:14.116006 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59"] Oct 30 20:58:14.119314 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:14.119295 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/distinct-2-w3-f6ebf5-kserve-65fbdc579-89p59"] Oct 30 20:58:14.230754 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:14.230727 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7"] Oct 30 20:58:14.238791 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:14.238764 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f6ebf5/premium-w3-f6ebf5-kserve-dfb898845-kvws7"] Oct 30 20:58:15.223259 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:15.223204 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="197b515b-ee40-4966-aa7f-f20e16001039" path="/var/lib/kubelet/pods/197b515b-ee40-4966-aa7f-f20e16001039/volumes" Oct 30 20:58:15.224679 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:15.223897 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" path="/var/lib/kubelet/pods/68b1092f-af0f-477c-81c0-cc3083b7f564/volumes" Oct 30 20:58:15.224679 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:15.224450 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" path="/var/lib/kubelet/pods/a305512d-f7b3-4b29-b72e-22115f7c8b0c/volumes" Oct 30 20:58:16.180211 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.180174 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg"] Oct 30 20:58:16.180756 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.180739 5755 cpu_manager.go:414] "containerMap: removing container" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" Oct 30 20:58:16.180806 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.180760 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" Oct 30 20:58:16.180806 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.180778 5755 cpu_manager.go:414] "containerMap: removing container" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" Oct 30 20:58:16.180806 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.180783 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" Oct 30 20:58:16.180806 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.180797 5755 cpu_manager.go:414] "containerMap: removing container" podUID="dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f" containerName="revoke-keys" Oct 30 20:58:16.180806 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.180802 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f" containerName="revoke-keys" Oct 30 20:58:16.180963 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.180829 5755 cpu_manager.go:414] "containerMap: removing container" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" Oct 30 20:58:16.180963 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.180837 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" Oct 30 20:58:16.180963 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.180919 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="197b515b-ee40-4966-aa7f-f20e16001039" containerName="main" Oct 30 20:58:16.180963 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.180927 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f" containerName="revoke-keys" Oct 30 20:58:16.180963 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.180938 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="a305512d-f7b3-4b29-b72e-22115f7c8b0c" containerName="main" Oct 30 20:58:16.180963 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.180947 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b1092f-af0f-477c-81c0-cc3083b7f564" containerName="main" Oct 30 20:58:16.185827 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.185810 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg" Oct 30 20:58:16.243323 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.243285 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg"] Oct 30 20:58:16.272093 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.272003 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg\" (UID: \"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg" Oct 30 20:58:16.272291 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.272128 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfk7s\" (UniqueName: \"kubernetes.io/projected/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-kube-api-access-qfk7s\") pod \"maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg\" (UID: \"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg" Oct 30 20:58:16.272291 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.272276 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg\" (UID: \"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg" Oct 30 20:58:16.289160 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.289138 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:58:16.289316 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.289302 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:58:16.374045 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.374014 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg\" (UID: \"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg" Oct 30 20:58:16.374215 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.374094 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfk7s\" (UniqueName: \"kubernetes.io/projected/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-kube-api-access-qfk7s\") pod \"maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg\" (UID: \"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg" Oct 30 20:58:16.374336 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.374309 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg\" (UID: \"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg" Oct 30 20:58:16.374776 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.374756 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg\" (UID: \"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg" Oct 30 20:58:16.376876 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.376854 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg\" (UID: \"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg" Oct 30 20:58:16.431962 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.431890 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfk7s\" (UniqueName: \"kubernetes.io/projected/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-kube-api-access-qfk7s\") pod \"maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg\" (UID: \"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg" Oct 30 20:58:16.498605 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.498577 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg" Oct 30 20:58:16.882763 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:16.882732 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg"] Oct 30 20:58:16.883228 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:58:16.883207 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf3fa41_c39a_4099_ae01_6ae6495a3bf6.slice/crio-3e08635df1bc81d22abd53cd07b3ba35e5457592dd54439ac2c32a97e1b8db92 WatchSource:0}: Error finding container 3e08635df1bc81d22abd53cd07b3ba35e5457592dd54439ac2c32a97e1b8db92: Status 404 returned error can't find the container with id 3e08635df1bc81d22abd53cd07b3ba35e5457592dd54439ac2c32a97e1b8db92 Oct 30 20:58:17.874678 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:17.874636 5755 generic.go:356] "Generic (PLEG): container finished" podID="bcf3fa41-c39a-4099-ae01-6ae6495a3bf6" containerID="81ce4cc9b21194ac01ab1c4c3df852cf4f50731cc9ae7707f3285a991d8d3b10" exitCode=0 Oct 30 20:58:17.875132 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:17.874721 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg" event={"ID":"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6","Type":"ContainerDied","Data":"81ce4cc9b21194ac01ab1c4c3df852cf4f50731cc9ae7707f3285a991d8d3b10"} Oct 30 20:58:17.875132 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:17.874793 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg" event={"ID":"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6","Type":"ContainerStarted","Data":"3e08635df1bc81d22abd53cd07b3ba35e5457592dd54439ac2c32a97e1b8db92"} Oct 30 20:58:19.034062 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:19.034040 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg" Oct 30 20:58:19.215872 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:19.215801 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-maas-api-openshift-ca\") pod \"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6\" (UID: \"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6\") " Oct 30 20:58:19.216001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:19.215881 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-maas-api-xks-ca\") pod \"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6\" (UID: \"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6\") " Oct 30 20:58:19.216084 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:19.216065 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-kube-api-access-qfk7s\" (UniqueName: \"kubernetes.io/projected/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-kube-api-access-qfk7s\") pod \"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6\" (UID: \"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6\") " Oct 30 20:58:19.216177 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:19.216159 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-maas-api-openshift-ca" pod "bcf3fa41-c39a-4099-ae01-6ae6495a3bf6" (UID: "bcf3fa41-c39a-4099-ae01-6ae6495a3bf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:19.216478 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:19.216455 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:19.218007 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:19.217985 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-maas-api-xks-ca" pod "bcf3fa41-c39a-4099-ae01-6ae6495a3bf6" (UID: "bcf3fa41-c39a-4099-ae01-6ae6495a3bf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:19.218128 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:19.218111 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-kube-api-access-qfk7s" pod "bcf3fa41-c39a-4099-ae01-6ae6495a3bf6" (UID: "bcf3fa41-c39a-4099-ae01-6ae6495a3bf6"). InnerVolumeSpecName "kube-api-access-qfk7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:19.317935 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:19.317893 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfk7s\" (UniqueName: \"kubernetes.io/projected/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-kube-api-access-qfk7s\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:19.317935 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:19.317928 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:19.889728 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:19.889698 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg" Oct 30 20:58:19.889923 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:19.889735 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg" event={"ID":"bcf3fa41-c39a-4099-ae01-6ae6495a3bf6","Type":"ContainerDied","Data":"3e08635df1bc81d22abd53cd07b3ba35e5457592dd54439ac2c32a97e1b8db92"} Oct 30 20:58:19.889923 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:19.889800 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e08635df1bc81d22abd53cd07b3ba35e5457592dd54439ac2c32a97e1b8db92" Oct 30 20:58:20.575213 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:20.575182 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg"] Oct 30 20:58:23.233848 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:23.233809 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" podUID="312ff01e-18e1-4d99-9dfd-67df3b8e640f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:58:30.305484 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:30.305418 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" Oct 30 20:58:30.377650 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:30.377610 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/312ff01e-18e1-4d99-9dfd-67df3b8e640f-kube-api-access-bkbnt\" (UniqueName: \"kubernetes.io/projected/312ff01e-18e1-4d99-9dfd-67df3b8e640f-kube-api-access-bkbnt\") pod \"312ff01e-18e1-4d99-9dfd-67df3b8e640f\" (UID: \"312ff01e-18e1-4d99-9dfd-67df3b8e640f\") " Oct 30 20:58:30.377808 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:30.377674 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/312ff01e-18e1-4d99-9dfd-67df3b8e640f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312ff01e-18e1-4d99-9dfd-67df3b8e640f-plugins-config-volume\") pod \"312ff01e-18e1-4d99-9dfd-67df3b8e640f\" (UID: \"312ff01e-18e1-4d99-9dfd-67df3b8e640f\") " Oct 30 20:58:30.378150 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:30.378117 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312ff01e-18e1-4d99-9dfd-67df3b8e640f-plugins-config-volume" pod "312ff01e-18e1-4d99-9dfd-67df3b8e640f" (UID: "312ff01e-18e1-4d99-9dfd-67df3b8e640f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:30.379939 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:30.379916 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312ff01e-18e1-4d99-9dfd-67df3b8e640f-kube-api-access-bkbnt" pod "312ff01e-18e1-4d99-9dfd-67df3b8e640f" (UID: "312ff01e-18e1-4d99-9dfd-67df3b8e640f"). InnerVolumeSpecName "kube-api-access-bkbnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:30.479680 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:30.479646 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkbnt\" (UniqueName: \"kubernetes.io/projected/312ff01e-18e1-4d99-9dfd-67df3b8e640f-kube-api-access-bkbnt\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:30.479680 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:30.479675 5755 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312ff01e-18e1-4d99-9dfd-67df3b8e640f-plugins-config-volume\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:30.974087 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:30.974048 5755 generic.go:356] "Generic (PLEG): container finished" podID="312ff01e-18e1-4d99-9dfd-67df3b8e640f" containerID="9a3703de5f02d5004b7709efc5cf47b6fd047f268008d977b1225b1229b02a39" exitCode=0 Oct 30 20:58:30.974292 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:30.974094 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" event={"ID":"312ff01e-18e1-4d99-9dfd-67df3b8e640f","Type":"ContainerDied","Data":"9a3703de5f02d5004b7709efc5cf47b6fd047f268008d977b1225b1229b02a39"} Oct 30 20:58:30.974292 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:30.974102 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" Oct 30 20:58:30.974292 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:30.974173 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn" event={"ID":"312ff01e-18e1-4d99-9dfd-67df3b8e640f","Type":"ContainerDied","Data":"586e7281791a6ef86130372fe766ff1822c2f56b7e52db383facf2d9f22769a6"} Oct 30 20:58:30.974292 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:30.974201 5755 scope.go:122] "RemoveContainer" containerID="9a3703de5f02d5004b7709efc5cf47b6fd047f268008d977b1225b1229b02a39" Oct 30 20:58:30.985175 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:30.985158 5755 scope.go:122] "RemoveContainer" containerID="9a3703de5f02d5004b7709efc5cf47b6fd047f268008d977b1225b1229b02a39" Oct 30 20:58:30.985454 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:58:30.985428 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a3703de5f02d5004b7709efc5cf47b6fd047f268008d977b1225b1229b02a39\": container with ID starting with 9a3703de5f02d5004b7709efc5cf47b6fd047f268008d977b1225b1229b02a39 not found: ID does not exist" containerID="9a3703de5f02d5004b7709efc5cf47b6fd047f268008d977b1225b1229b02a39" Oct 30 20:58:30.985506 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:30.985463 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a3703de5f02d5004b7709efc5cf47b6fd047f268008d977b1225b1229b02a39"} err="failed to get container status \"9a3703de5f02d5004b7709efc5cf47b6fd047f268008d977b1225b1229b02a39\": rpc error: code = NotFound desc = could not find container \"9a3703de5f02d5004b7709efc5cf47b6fd047f268008d977b1225b1229b02a39\": container with ID starting with 9a3703de5f02d5004b7709efc5cf47b6fd047f268008d977b1225b1229b02a39 not found: ID does not exist" Oct 30 20:58:31.144355 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:31.144313 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn"] Oct 30 20:58:31.192974 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:31.192946 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f6ebf5-fc4cfcfcb-7qfjn"] Oct 30 20:58:31.212114 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:31.212085 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="312ff01e-18e1-4d99-9dfd-67df3b8e640f" path="/var/lib/kubelet/pods/312ff01e-18e1-4d99-9dfd-67df3b8e640f/volumes" Oct 30 20:58:35.596373 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.596315 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" containerID="cri-o://1a1e777669ec876b4b0e944b38b34bb5c4e04994185e100d768974505fb12b16" gracePeriod=45 Oct 30 20:58:35.861243 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.861189 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:58:35.967886 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.967857 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-tmp-dir\") pod \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " Oct 30 20:58:35.968053 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.967926 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e57bd5-935c-4aac-b511-bbfd0068ebb8-kube-api-access-9twdk\" (UniqueName: \"kubernetes.io/projected/51e57bd5-935c-4aac-b511-bbfd0068ebb8-kube-api-access-9twdk\") pod \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " Oct 30 20:58:35.968053 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.968040 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-dshm\") pod \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " Oct 30 20:58:35.968147 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.968081 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-home\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-home\") pod \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " Oct 30 20:58:35.968147 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.968124 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-model-cache\") pod \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " Oct 30 20:58:35.968147 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.968125 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-tmp-dir" pod "51e57bd5-935c-4aac-b511-bbfd0068ebb8" (UID: "51e57bd5-935c-4aac-b511-bbfd0068ebb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:35.968262 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.968154 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51e57bd5-935c-4aac-b511-bbfd0068ebb8-tls-certs\" (UniqueName: \"kubernetes.io/secret/51e57bd5-935c-4aac-b511-bbfd0068ebb8-tls-certs\") pod \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\" (UID: \"51e57bd5-935c-4aac-b511-bbfd0068ebb8\") " Oct 30 20:58:35.968356 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.968332 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-home" pod "51e57bd5-935c-4aac-b511-bbfd0068ebb8" (UID: "51e57bd5-935c-4aac-b511-bbfd0068ebb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:35.968356 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.968343 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-model-cache" pod "51e57bd5-935c-4aac-b511-bbfd0068ebb8" (UID: "51e57bd5-935c-4aac-b511-bbfd0068ebb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:35.968796 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.968775 5755 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-tmp-dir\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:35.968889 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.968801 5755 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-home\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:35.968889 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.968815 5755 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-model-cache\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:35.970318 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.970296 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-dshm" pod "51e57bd5-935c-4aac-b511-bbfd0068ebb8" (UID: "51e57bd5-935c-4aac-b511-bbfd0068ebb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:35.970434 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.970372 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e57bd5-935c-4aac-b511-bbfd0068ebb8-kube-api-access-9twdk" pod "51e57bd5-935c-4aac-b511-bbfd0068ebb8" (UID: "51e57bd5-935c-4aac-b511-bbfd0068ebb8"). InnerVolumeSpecName "kube-api-access-9twdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:35.970490 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:35.970439 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e57bd5-935c-4aac-b511-bbfd0068ebb8-tls-certs" pod "51e57bd5-935c-4aac-b511-bbfd0068ebb8" (UID: "51e57bd5-935c-4aac-b511-bbfd0068ebb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:36.030588 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:36.030555 5755 generic.go:356] "Generic (PLEG): container finished" podID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerID="1a1e777669ec876b4b0e944b38b34bb5c4e04994185e100d768974505fb12b16" exitCode=0 Oct 30 20:58:36.030743 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:36.030613 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" Oct 30 20:58:36.030743 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:36.030634 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" event={"ID":"51e57bd5-935c-4aac-b511-bbfd0068ebb8","Type":"ContainerDied","Data":"1a1e777669ec876b4b0e944b38b34bb5c4e04994185e100d768974505fb12b16"} Oct 30 20:58:36.030743 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:36.030678 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg" event={"ID":"51e57bd5-935c-4aac-b511-bbfd0068ebb8","Type":"ContainerDied","Data":"16298d11d6dbf3e95dfcaf7d399712f4447d659ccc0700a6dbc0a1be11f2775e"} Oct 30 20:58:36.030743 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:36.030695 5755 scope.go:122] "RemoveContainer" containerID="1a1e777669ec876b4b0e944b38b34bb5c4e04994185e100d768974505fb12b16" Oct 30 20:58:36.041895 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:36.041877 5755 scope.go:122] "RemoveContainer" containerID="1a1e777669ec876b4b0e944b38b34bb5c4e04994185e100d768974505fb12b16" Oct 30 20:58:36.042205 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:58:36.042188 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1e777669ec876b4b0e944b38b34bb5c4e04994185e100d768974505fb12b16\": container with ID starting with 1a1e777669ec876b4b0e944b38b34bb5c4e04994185e100d768974505fb12b16 not found: ID does not exist" containerID="1a1e777669ec876b4b0e944b38b34bb5c4e04994185e100d768974505fb12b16" Oct 30 20:58:36.042248 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:36.042215 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1e777669ec876b4b0e944b38b34bb5c4e04994185e100d768974505fb12b16"} err="failed to get container status \"1a1e777669ec876b4b0e944b38b34bb5c4e04994185e100d768974505fb12b16\": rpc error: code = NotFound desc = could not find container \"1a1e777669ec876b4b0e944b38b34bb5c4e04994185e100d768974505fb12b16\": container with ID starting with 1a1e777669ec876b4b0e944b38b34bb5c4e04994185e100d768974505fb12b16 not found: ID does not exist" Oct 30 20:58:36.070438 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:36.070411 5755 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e57bd5-935c-4aac-b511-bbfd0068ebb8-tls-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:36.070568 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:36.070441 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9twdk\" (UniqueName: \"kubernetes.io/projected/51e57bd5-935c-4aac-b511-bbfd0068ebb8-kube-api-access-9twdk\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:36.070568 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:36.070460 5755 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e57bd5-935c-4aac-b511-bbfd0068ebb8-dshm\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:36.201458 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:36.201357 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg"] Oct 30 20:58:36.206454 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:36.206427 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b43c84/distinct-2-w2-b43c84-kserve-76fb6865bd-m97mg"] Oct 30 20:58:37.219304 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:37.219265 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" path="/var/lib/kubelet/pods/51e57bd5-935c-4aac-b511-bbfd0068ebb8/volumes" Oct 30 20:58:39.241007 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.240972 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq"] Oct 30 20:58:39.241472 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.241457 5755 cpu_manager.go:414] "containerMap: removing container" podUID="bcf3fa41-c39a-4099-ae01-6ae6495a3bf6" containerName="revoke-keys" Oct 30 20:58:39.241525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.241473 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf3fa41-c39a-4099-ae01-6ae6495a3bf6" containerName="revoke-keys" Oct 30 20:58:39.241525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.241494 5755 cpu_manager.go:414] "containerMap: removing container" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" Oct 30 20:58:39.241525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.241499 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" Oct 30 20:58:39.241525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.241517 5755 cpu_manager.go:414] "containerMap: removing container" podUID="312ff01e-18e1-4d99-9dfd-67df3b8e640f" containerName="payload-processing" Oct 30 20:58:39.241525 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.241522 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="312ff01e-18e1-4d99-9dfd-67df3b8e640f" containerName="payload-processing" Oct 30 20:58:39.241679 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.241596 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="312ff01e-18e1-4d99-9dfd-67df3b8e640f" containerName="payload-processing" Oct 30 20:58:39.241679 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.241607 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf3fa41-c39a-4099-ae01-6ae6495a3bf6" containerName="revoke-keys" Oct 30 20:58:39.241679 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.241615 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e57bd5-935c-4aac-b511-bbfd0068ebb8" containerName="main" Oct 30 20:58:39.244811 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.244794 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" Oct 30 20:58:39.301705 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.301679 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq"] Oct 30 20:58:39.347614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.347584 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:58:39.347614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.347597 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:58:39.412728 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.412692 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qzcl\" (UniqueName: \"kubernetes.io/projected/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-kube-api-access-9qzcl\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq\" (UID: \"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" Oct 30 20:58:39.412888 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.412751 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq\" (UID: \"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" Oct 30 20:58:39.412934 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.412921 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq\" (UID: \"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" Oct 30 20:58:39.530465 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.519162 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq\" (UID: \"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" Oct 30 20:58:39.530465 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.519605 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qzcl\" (UniqueName: \"kubernetes.io/projected/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-kube-api-access-9qzcl\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq\" (UID: \"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" Oct 30 20:58:39.530465 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.519742 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq\" (UID: \"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" Oct 30 20:58:39.535894 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.535860 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq\" (UID: \"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" Oct 30 20:58:39.585431 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.585398 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq\" (UID: \"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" Oct 30 20:58:39.588435 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.588409 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qzcl\" (UniqueName: \"kubernetes.io/projected/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-kube-api-access-9qzcl\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq\" (UID: \"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" Oct 30 20:58:39.859286 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:39.859209 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" Oct 30 20:58:40.251530 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:58:40.251495 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfc7d29_6fed_4ef9_be4f_b30d6a59c069.slice/crio-90d8bba62cc43e220b2c78e371d8fb7cbab8d593e61dea7eccf0b550351366d8 WatchSource:0}: Error finding container 90d8bba62cc43e220b2c78e371d8fb7cbab8d593e61dea7eccf0b550351366d8: Status 404 returned error can't find the container with id 90d8bba62cc43e220b2c78e371d8fb7cbab8d593e61dea7eccf0b550351366d8 Oct 30 20:58:40.251868 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:40.251606 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq"] Oct 30 20:58:40.818749 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:40.818714 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz"] Oct 30 20:58:40.861633 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:40.861556 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-15199a-8b9tz"] Oct 30 20:58:41.080098 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:41.080059 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" event={"ID":"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069","Type":"ContainerStarted","Data":"7b713b8eb95497abf842e177aa9821e851f4b921472b3cef32bc02126d398878"} Oct 30 20:58:41.080270 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:41.080141 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" event={"ID":"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069","Type":"ContainerStarted","Data":"90d8bba62cc43e220b2c78e371d8fb7cbab8d593e61dea7eccf0b550351366d8"} Oct 30 20:58:41.202730 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:41.202625 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" podStartSLOduration=2.202610809 podStartE2EDuration="2.202610809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:41.198430285 +0000 UTC m=+1884.124987200" watchObservedRunningTime="2026-10-30 20:58:41.202610809 +0000 UTC m=+1884.129167741" Oct 30 20:58:41.212981 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:41.212949 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18683c1f-ed8e-4a35-be3b-7ef8b3795e6e" path="/var/lib/kubelet/pods/18683c1f-ed8e-4a35-be3b-7ef8b3795e6e/volumes" Oct 30 20:58:41.258127 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:58:41.258095 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:58:42.087568 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:42.087536 5755 generic.go:356] "Generic (PLEG): container finished" podID="8cfc7d29-6fed-4ef9-be4f-b30d6a59c069" containerID="7b713b8eb95497abf842e177aa9821e851f4b921472b3cef32bc02126d398878" exitCode=0 Oct 30 20:58:42.087759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:42.087664 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" event={"ID":"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069","Type":"ContainerDied","Data":"7b713b8eb95497abf842e177aa9821e851f4b921472b3cef32bc02126d398878"} Oct 30 20:58:43.231731 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:43.231708 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" Oct 30 20:58:43.267975 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:43.267949 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-kube-api-access-9qzcl\" (UniqueName: \"kubernetes.io/projected/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-kube-api-access-9qzcl\") pod \"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069\" (UID: \"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069\") " Oct 30 20:58:43.268127 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:43.267981 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-maas-api-xks-ca\") pod \"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069\" (UID: \"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069\") " Oct 30 20:58:43.268127 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:43.268006 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-maas-api-openshift-ca\") pod \"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069\" (UID: \"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069\") " Oct 30 20:58:43.268362 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:43.268341 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-maas-api-openshift-ca" pod "8cfc7d29-6fed-4ef9-be4f-b30d6a59c069" (UID: "8cfc7d29-6fed-4ef9-be4f-b30d6a59c069"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:43.268551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:43.268535 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:43.270128 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:43.270106 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-maas-api-xks-ca" pod "8cfc7d29-6fed-4ef9-be4f-b30d6a59c069" (UID: "8cfc7d29-6fed-4ef9-be4f-b30d6a59c069"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:43.270271 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:43.270251 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-kube-api-access-9qzcl" pod "8cfc7d29-6fed-4ef9-be4f-b30d6a59c069" (UID: "8cfc7d29-6fed-4ef9-be4f-b30d6a59c069"). InnerVolumeSpecName "kube-api-access-9qzcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:43.369878 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:43.369808 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qzcl\" (UniqueName: \"kubernetes.io/projected/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-kube-api-access-9qzcl\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:43.369878 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:43.369837 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:44.101841 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:44.101808 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" Oct 30 20:58:44.101841 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:44.101817 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq" event={"ID":"8cfc7d29-6fed-4ef9-be4f-b30d6a59c069","Type":"ContainerDied","Data":"90d8bba62cc43e220b2c78e371d8fb7cbab8d593e61dea7eccf0b550351366d8"} Oct 30 20:58:44.102049 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:44.101861 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90d8bba62cc43e220b2c78e371d8fb7cbab8d593e61dea7eccf0b550351366d8" Oct 30 20:58:45.397972 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.397931 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s"] Oct 30 20:58:45.398523 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.398504 5755 cpu_manager.go:414] "containerMap: removing container" podUID="8cfc7d29-6fed-4ef9-be4f-b30d6a59c069" containerName="revoke-keys" Oct 30 20:58:45.398523 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.398523 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cfc7d29-6fed-4ef9-be4f-b30d6a59c069" containerName="revoke-keys" Oct 30 20:58:45.398641 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.398612 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cfc7d29-6fed-4ef9-be4f-b30d6a59c069" containerName="revoke-keys" Oct 30 20:58:45.402426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.402404 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s" Oct 30 20:58:45.453877 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.453848 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp"] Oct 30 20:58:45.458036 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.458007 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp" Oct 30 20:58:45.461252 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.461223 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s"] Oct 30 20:58:45.498734 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.498703 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a12be28-f400-43ee-bb32-1c612e686542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90960b22-nfh2s\" (UID: \"3a12be28-f400-43ee-bb32-1c612e686542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s" Oct 30 20:58:45.498917 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.498871 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a12be28-f400-43ee-bb32-1c612e686542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90960b22-nfh2s\" (UID: \"3a12be28-f400-43ee-bb32-1c612e686542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s" Oct 30 20:58:45.498992 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.498919 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhtfw\" (UniqueName: \"kubernetes.io/projected/3a12be28-f400-43ee-bb32-1c612e686542-kube-api-access-qhtfw\") pod \"maas-api-revoke-sub-90960b22-nfh2s\" (UID: \"3a12be28-f400-43ee-bb32-1c612e686542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s" Oct 30 20:58:45.498992 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.498941 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmf7x\" (UniqueName: \"kubernetes.io/projected/18314524-5ec3-4f96-8a26-8696f8b1fbd3-kube-api-access-nmf7x\") pod \"maas-api-revoke-sub-f36ec720-f52dp\" (UID: \"18314524-5ec3-4f96-8a26-8696f8b1fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp" Oct 30 20:58:45.499114 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.499070 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18314524-5ec3-4f96-8a26-8696f8b1fbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f36ec720-f52dp\" (UID: \"18314524-5ec3-4f96-8a26-8696f8b1fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp" Oct 30 20:58:45.499169 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.499117 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18314524-5ec3-4f96-8a26-8696f8b1fbd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f36ec720-f52dp\" (UID: \"18314524-5ec3-4f96-8a26-8696f8b1fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp" Oct 30 20:58:45.499841 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.499808 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:58:45.499841 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.499831 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:58:45.515013 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.514985 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp"] Oct 30 20:58:45.600683 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.600649 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18314524-5ec3-4f96-8a26-8696f8b1fbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f36ec720-f52dp\" (UID: \"18314524-5ec3-4f96-8a26-8696f8b1fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp" Oct 30 20:58:45.600891 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.600694 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18314524-5ec3-4f96-8a26-8696f8b1fbd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f36ec720-f52dp\" (UID: \"18314524-5ec3-4f96-8a26-8696f8b1fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp" Oct 30 20:58:45.600891 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.600820 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a12be28-f400-43ee-bb32-1c612e686542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90960b22-nfh2s\" (UID: \"3a12be28-f400-43ee-bb32-1c612e686542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s" Oct 30 20:58:45.601017 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.600979 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a12be28-f400-43ee-bb32-1c612e686542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90960b22-nfh2s\" (UID: \"3a12be28-f400-43ee-bb32-1c612e686542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s" Oct 30 20:58:45.601074 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.601033 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhtfw\" (UniqueName: \"kubernetes.io/projected/3a12be28-f400-43ee-bb32-1c612e686542-kube-api-access-qhtfw\") pod \"maas-api-revoke-sub-90960b22-nfh2s\" (UID: \"3a12be28-f400-43ee-bb32-1c612e686542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s" Oct 30 20:58:45.601074 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.601058 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmf7x\" (UniqueName: \"kubernetes.io/projected/18314524-5ec3-4f96-8a26-8696f8b1fbd3-kube-api-access-nmf7x\") pod \"maas-api-revoke-sub-f36ec720-f52dp\" (UID: \"18314524-5ec3-4f96-8a26-8696f8b1fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp" Oct 30 20:58:45.601489 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.601449 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18314524-5ec3-4f96-8a26-8696f8b1fbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f36ec720-f52dp\" (UID: \"18314524-5ec3-4f96-8a26-8696f8b1fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp" Oct 30 20:58:45.601773 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.601623 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a12be28-f400-43ee-bb32-1c612e686542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90960b22-nfh2s\" (UID: \"3a12be28-f400-43ee-bb32-1c612e686542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s" Oct 30 20:58:45.603246 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.603225 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18314524-5ec3-4f96-8a26-8696f8b1fbd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f36ec720-f52dp\" (UID: \"18314524-5ec3-4f96-8a26-8696f8b1fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp" Oct 30 20:58:45.603456 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.603437 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a12be28-f400-43ee-bb32-1c612e686542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90960b22-nfh2s\" (UID: \"3a12be28-f400-43ee-bb32-1c612e686542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s" Oct 30 20:58:45.661146 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.661075 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhtfw\" (UniqueName: \"kubernetes.io/projected/3a12be28-f400-43ee-bb32-1c612e686542-kube-api-access-qhtfw\") pod \"maas-api-revoke-sub-90960b22-nfh2s\" (UID: \"3a12be28-f400-43ee-bb32-1c612e686542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s" Oct 30 20:58:45.661146 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.661138 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmf7x\" (UniqueName: \"kubernetes.io/projected/18314524-5ec3-4f96-8a26-8696f8b1fbd3-kube-api-access-nmf7x\") pod \"maas-api-revoke-sub-f36ec720-f52dp\" (UID: \"18314524-5ec3-4f96-8a26-8696f8b1fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp" Oct 30 20:58:45.715833 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.715797 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s" Oct 30 20:58:45.773263 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:45.773233 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp" Oct 30 20:58:46.093013 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:46.092979 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s"] Oct 30 20:58:46.093802 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:58:46.093766 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a12be28_f400_43ee_bb32_1c612e686542.slice/crio-d9c27e794caeb8109b8b1c4d18e9a0cfe68c0c02e3e4df92259ff1c62c95eb8b WatchSource:0}: Error finding container d9c27e794caeb8109b8b1c4d18e9a0cfe68c0c02e3e4df92259ff1c62c95eb8b: Status 404 returned error can't find the container with id d9c27e794caeb8109b8b1c4d18e9a0cfe68c0c02e3e4df92259ff1c62c95eb8b Oct 30 20:58:46.116746 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:46.116720 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s" event={"ID":"3a12be28-f400-43ee-bb32-1c612e686542","Type":"ContainerStarted","Data":"d9c27e794caeb8109b8b1c4d18e9a0cfe68c0c02e3e4df92259ff1c62c95eb8b"} Oct 30 20:58:46.202953 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:46.202931 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp"] Oct 30 20:58:46.203459 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:58:46.203431 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18314524_5ec3_4f96_8a26_8696f8b1fbd3.slice/crio-2cafc883ac550c906ef82e8b1ffc73a3a513757be63a6b59d81231691d365330 WatchSource:0}: Error finding container 2cafc883ac550c906ef82e8b1ffc73a3a513757be63a6b59d81231691d365330: Status 404 returned error can't find the container with id 2cafc883ac550c906ef82e8b1ffc73a3a513757be63a6b59d81231691d365330 Oct 30 20:58:47.124640 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:47.124607 5755 generic.go:356] "Generic (PLEG): container finished" podID="18314524-5ec3-4f96-8a26-8696f8b1fbd3" containerID="a6f8b190c38ce2401098777f72e24b261d6ad23f4a6ad2d815df6820db78c534" exitCode=0 Oct 30 20:58:47.125022 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:47.124701 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp" event={"ID":"18314524-5ec3-4f96-8a26-8696f8b1fbd3","Type":"ContainerDied","Data":"a6f8b190c38ce2401098777f72e24b261d6ad23f4a6ad2d815df6820db78c534"} Oct 30 20:58:47.125022 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:47.124809 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp" event={"ID":"18314524-5ec3-4f96-8a26-8696f8b1fbd3","Type":"ContainerStarted","Data":"2cafc883ac550c906ef82e8b1ffc73a3a513757be63a6b59d81231691d365330"} Oct 30 20:58:47.126304 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:47.126281 5755 generic.go:356] "Generic (PLEG): container finished" podID="3a12be28-f400-43ee-bb32-1c612e686542" containerID="2e69bc345a47e59699c054fe693a7d2daf35fe0fd38f4d0b74e18e34edc03aab" exitCode=0 Oct 30 20:58:47.126432 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:47.126349 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s" event={"ID":"3a12be28-f400-43ee-bb32-1c612e686542","Type":"ContainerDied","Data":"2e69bc345a47e59699c054fe693a7d2daf35fe0fd38f4d0b74e18e34edc03aab"} Oct 30 20:58:48.316906 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.316877 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s" Oct 30 20:58:48.330215 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.330195 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp" Oct 30 20:58:48.432444 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.432331 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18314524-5ec3-4f96-8a26-8696f8b1fbd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18314524-5ec3-4f96-8a26-8696f8b1fbd3-maas-api-openshift-ca\") pod \"18314524-5ec3-4f96-8a26-8696f8b1fbd3\" (UID: \"18314524-5ec3-4f96-8a26-8696f8b1fbd3\") " Oct 30 20:58:48.432444 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.432367 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18314524-5ec3-4f96-8a26-8696f8b1fbd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18314524-5ec3-4f96-8a26-8696f8b1fbd3-maas-api-xks-ca\") pod \"18314524-5ec3-4f96-8a26-8696f8b1fbd3\" (UID: \"18314524-5ec3-4f96-8a26-8696f8b1fbd3\") " Oct 30 20:58:48.432669 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.432462 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a12be28-f400-43ee-bb32-1c612e686542-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a12be28-f400-43ee-bb32-1c612e686542-maas-api-openshift-ca\") pod \"3a12be28-f400-43ee-bb32-1c612e686542\" (UID: \"3a12be28-f400-43ee-bb32-1c612e686542\") " Oct 30 20:58:48.432669 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.432647 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18314524-5ec3-4f96-8a26-8696f8b1fbd3-kube-api-access-nmf7x\" (UniqueName: \"kubernetes.io/projected/18314524-5ec3-4f96-8a26-8696f8b1fbd3-kube-api-access-nmf7x\") pod \"18314524-5ec3-4f96-8a26-8696f8b1fbd3\" (UID: \"18314524-5ec3-4f96-8a26-8696f8b1fbd3\") " Oct 30 20:58:48.432991 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.432966 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a12be28-f400-43ee-bb32-1c612e686542-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a12be28-f400-43ee-bb32-1c612e686542-maas-api-xks-ca\") pod \"3a12be28-f400-43ee-bb32-1c612e686542\" (UID: \"3a12be28-f400-43ee-bb32-1c612e686542\") " Oct 30 20:58:48.433098 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.433032 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a12be28-f400-43ee-bb32-1c612e686542-kube-api-access-qhtfw\" (UniqueName: \"kubernetes.io/projected/3a12be28-f400-43ee-bb32-1c612e686542-kube-api-access-qhtfw\") pod \"3a12be28-f400-43ee-bb32-1c612e686542\" (UID: \"3a12be28-f400-43ee-bb32-1c612e686542\") " Oct 30 20:58:48.434210 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.434180 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a12be28-f400-43ee-bb32-1c612e686542-maas-api-openshift-ca" pod "3a12be28-f400-43ee-bb32-1c612e686542" (UID: "3a12be28-f400-43ee-bb32-1c612e686542"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:48.435797 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.435772 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18314524-5ec3-4f96-8a26-8696f8b1fbd3-maas-api-openshift-ca" pod "18314524-5ec3-4f96-8a26-8696f8b1fbd3" (UID: "18314524-5ec3-4f96-8a26-8696f8b1fbd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:48.436103 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.436085 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18314524-5ec3-4f96-8a26-8696f8b1fbd3-kube-api-access-nmf7x" pod "18314524-5ec3-4f96-8a26-8696f8b1fbd3" (UID: "18314524-5ec3-4f96-8a26-8696f8b1fbd3"). InnerVolumeSpecName "kube-api-access-nmf7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:48.437879 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.437851 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a12be28-f400-43ee-bb32-1c612e686542-maas-api-xks-ca" pod "3a12be28-f400-43ee-bb32-1c612e686542" (UID: "3a12be28-f400-43ee-bb32-1c612e686542"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:48.438232 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.438202 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18314524-5ec3-4f96-8a26-8696f8b1fbd3-maas-api-xks-ca" pod "18314524-5ec3-4f96-8a26-8696f8b1fbd3" (UID: "18314524-5ec3-4f96-8a26-8696f8b1fbd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:48.438531 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.438502 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a12be28-f400-43ee-bb32-1c612e686542-kube-api-access-qhtfw" pod "3a12be28-f400-43ee-bb32-1c612e686542" (UID: "3a12be28-f400-43ee-bb32-1c612e686542"). InnerVolumeSpecName "kube-api-access-qhtfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:48.534929 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.534899 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmf7x\" (UniqueName: \"kubernetes.io/projected/18314524-5ec3-4f96-8a26-8696f8b1fbd3-kube-api-access-nmf7x\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:48.534929 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.534925 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a12be28-f400-43ee-bb32-1c612e686542-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:48.535103 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.534936 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhtfw\" (UniqueName: \"kubernetes.io/projected/3a12be28-f400-43ee-bb32-1c612e686542-kube-api-access-qhtfw\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:48.535103 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.534945 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18314524-5ec3-4f96-8a26-8696f8b1fbd3-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:48.535103 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.534954 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18314524-5ec3-4f96-8a26-8696f8b1fbd3-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:48.535103 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:48.534963 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a12be28-f400-43ee-bb32-1c612e686542-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:49.142370 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:49.142339 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s" Oct 30 20:58:49.142578 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:49.142358 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s" event={"ID":"3a12be28-f400-43ee-bb32-1c612e686542","Type":"ContainerDied","Data":"d9c27e794caeb8109b8b1c4d18e9a0cfe68c0c02e3e4df92259ff1c62c95eb8b"} Oct 30 20:58:49.142578 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:49.142460 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9c27e794caeb8109b8b1c4d18e9a0cfe68c0c02e3e4df92259ff1c62c95eb8b" Oct 30 20:58:49.143860 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:49.143829 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp" Oct 30 20:58:49.143860 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:49.143845 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp" event={"ID":"18314524-5ec3-4f96-8a26-8696f8b1fbd3","Type":"ContainerDied","Data":"2cafc883ac550c906ef82e8b1ffc73a3a513757be63a6b59d81231691d365330"} Oct 30 20:58:49.144074 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:49.143886 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cafc883ac550c906ef82e8b1ffc73a3a513757be63a6b59d81231691d365330" Oct 30 20:58:50.210712 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.210683 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj"] Oct 30 20:58:50.211182 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.211164 5755 cpu_manager.go:414] "containerMap: removing container" podUID="18314524-5ec3-4f96-8a26-8696f8b1fbd3" containerName="revoke-keys" Oct 30 20:58:50.211227 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.211205 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18314524-5ec3-4f96-8a26-8696f8b1fbd3" containerName="revoke-keys" Oct 30 20:58:50.211227 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.211215 5755 cpu_manager.go:414] "containerMap: removing container" podUID="3a12be28-f400-43ee-bb32-1c612e686542" containerName="revoke-keys" Oct 30 20:58:50.211227 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.211221 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a12be28-f400-43ee-bb32-1c612e686542" containerName="revoke-keys" Oct 30 20:58:50.211329 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.211320 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="18314524-5ec3-4f96-8a26-8696f8b1fbd3" containerName="revoke-keys" Oct 30 20:58:50.211364 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.211330 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a12be28-f400-43ee-bb32-1c612e686542" containerName="revoke-keys" Oct 30 20:58:50.214850 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.214827 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj" Oct 30 20:58:50.267446 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.267381 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d186b-4be6-4e85-9614-495004fa6ab2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-386743-mpplj\" (UID: \"ec2d186b-4be6-4e85-9614-495004fa6ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj" Oct 30 20:58:50.267661 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.267646 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrknn\" (UniqueName: \"kubernetes.io/projected/ec2d186b-4be6-4e85-9614-495004fa6ab2-kube-api-access-mrknn\") pod \"maas-api-revoke-keys-e2e-worker-w5-386743-mpplj\" (UID: \"ec2d186b-4be6-4e85-9614-495004fa6ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj" Oct 30 20:58:50.267759 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.267690 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2d186b-4be6-4e85-9614-495004fa6ab2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-386743-mpplj\" (UID: \"ec2d186b-4be6-4e85-9614-495004fa6ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj" Oct 30 20:58:50.271959 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.271935 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj"] Oct 30 20:58:50.317102 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.317078 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:58:50.317247 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.317162 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:58:50.368896 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.368870 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrknn\" (UniqueName: \"kubernetes.io/projected/ec2d186b-4be6-4e85-9614-495004fa6ab2-kube-api-access-mrknn\") pod \"maas-api-revoke-keys-e2e-worker-w5-386743-mpplj\" (UID: \"ec2d186b-4be6-4e85-9614-495004fa6ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj" Oct 30 20:58:50.369036 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.368921 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2d186b-4be6-4e85-9614-495004fa6ab2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-386743-mpplj\" (UID: \"ec2d186b-4be6-4e85-9614-495004fa6ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj" Oct 30 20:58:50.369036 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.368998 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d186b-4be6-4e85-9614-495004fa6ab2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-386743-mpplj\" (UID: \"ec2d186b-4be6-4e85-9614-495004fa6ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj" Oct 30 20:58:50.369708 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.369686 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d186b-4be6-4e85-9614-495004fa6ab2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-386743-mpplj\" (UID: \"ec2d186b-4be6-4e85-9614-495004fa6ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj" Oct 30 20:58:50.371441 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.371419 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2d186b-4be6-4e85-9614-495004fa6ab2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-386743-mpplj\" (UID: \"ec2d186b-4be6-4e85-9614-495004fa6ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj" Oct 30 20:58:50.428325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.428286 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrknn\" (UniqueName: \"kubernetes.io/projected/ec2d186b-4be6-4e85-9614-495004fa6ab2-kube-api-access-mrknn\") pod \"maas-api-revoke-keys-e2e-worker-w5-386743-mpplj\" (UID: \"ec2d186b-4be6-4e85-9614-495004fa6ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj" Oct 30 20:58:50.529212 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.527895 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj" Oct 30 20:58:50.967731 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:50.967704 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj"] Oct 30 20:58:50.968722 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:58:50.968689 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2d186b_4be6_4e85_9614_495004fa6ab2.slice/crio-8149c7a7bc7586df0bb138bd1ce582a85e358bd5e1abde7acb789670095f7c5f WatchSource:0}: Error finding container 8149c7a7bc7586df0bb138bd1ce582a85e358bd5e1abde7acb789670095f7c5f: Status 404 returned error can't find the container with id 8149c7a7bc7586df0bb138bd1ce582a85e358bd5e1abde7acb789670095f7c5f Oct 30 20:58:51.159324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.159284 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj" event={"ID":"ec2d186b-4be6-4e85-9614-495004fa6ab2","Type":"ContainerStarted","Data":"8149c7a7bc7586df0bb138bd1ce582a85e358bd5e1abde7acb789670095f7c5f"} Oct 30 20:58:51.504115 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.504081 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww"] Oct 30 20:58:51.507936 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.507916 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww" Oct 30 20:58:51.556988 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.556912 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2"] Oct 30 20:58:51.560823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.560802 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" Oct 30 20:58:51.568082 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.568056 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww"] Oct 30 20:58:51.588775 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.588745 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ad5871-dc94-4512-8f9b-3a754d439dca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cacd0ea-b68s2\" (UID: \"54ad5871-dc94-4512-8f9b-3a754d439dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" Oct 30 20:58:51.588915 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.588897 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/499437e4-d0c8-4bb9-972f-9843b0233962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1781e14-pkpww\" (UID: \"499437e4-d0c8-4bb9-972f-9843b0233962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww" Oct 30 20:58:51.588983 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.588955 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsgbw\" (UniqueName: \"kubernetes.io/projected/54ad5871-dc94-4512-8f9b-3a754d439dca-kube-api-access-zsgbw\") pod \"maas-api-revoke-sub-1cacd0ea-b68s2\" (UID: \"54ad5871-dc94-4512-8f9b-3a754d439dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" Oct 30 20:58:51.589029 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.588984 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ad5871-dc94-4512-8f9b-3a754d439dca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cacd0ea-b68s2\" (UID: \"54ad5871-dc94-4512-8f9b-3a754d439dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" Oct 30 20:58:51.589192 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.589170 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9mwq\" (UniqueName: \"kubernetes.io/projected/499437e4-d0c8-4bb9-972f-9843b0233962-kube-api-access-d9mwq\") pod \"maas-api-revoke-sub-b1781e14-pkpww\" (UID: \"499437e4-d0c8-4bb9-972f-9843b0233962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww" Oct 30 20:58:51.589231 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.589204 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/499437e4-d0c8-4bb9-972f-9843b0233962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1781e14-pkpww\" (UID: \"499437e4-d0c8-4bb9-972f-9843b0233962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww" Oct 30 20:58:51.618798 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.618767 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2"] Oct 30 20:58:51.690168 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.690135 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9mwq\" (UniqueName: \"kubernetes.io/projected/499437e4-d0c8-4bb9-972f-9843b0233962-kube-api-access-d9mwq\") pod \"maas-api-revoke-sub-b1781e14-pkpww\" (UID: \"499437e4-d0c8-4bb9-972f-9843b0233962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww" Oct 30 20:58:51.690168 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.690173 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/499437e4-d0c8-4bb9-972f-9843b0233962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1781e14-pkpww\" (UID: \"499437e4-d0c8-4bb9-972f-9843b0233962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww" Oct 30 20:58:51.690438 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.690202 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ad5871-dc94-4512-8f9b-3a754d439dca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cacd0ea-b68s2\" (UID: \"54ad5871-dc94-4512-8f9b-3a754d439dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" Oct 30 20:58:51.690438 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.690255 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/499437e4-d0c8-4bb9-972f-9843b0233962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1781e14-pkpww\" (UID: \"499437e4-d0c8-4bb9-972f-9843b0233962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww" Oct 30 20:58:51.690438 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.690288 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsgbw\" (UniqueName: \"kubernetes.io/projected/54ad5871-dc94-4512-8f9b-3a754d439dca-kube-api-access-zsgbw\") pod \"maas-api-revoke-sub-1cacd0ea-b68s2\" (UID: \"54ad5871-dc94-4512-8f9b-3a754d439dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" Oct 30 20:58:51.690438 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.690309 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ad5871-dc94-4512-8f9b-3a754d439dca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cacd0ea-b68s2\" (UID: \"54ad5871-dc94-4512-8f9b-3a754d439dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" Oct 30 20:58:51.690964 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.690937 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/499437e4-d0c8-4bb9-972f-9843b0233962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1781e14-pkpww\" (UID: \"499437e4-d0c8-4bb9-972f-9843b0233962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww" Oct 30 20:58:51.691078 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.691018 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ad5871-dc94-4512-8f9b-3a754d439dca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cacd0ea-b68s2\" (UID: \"54ad5871-dc94-4512-8f9b-3a754d439dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" Oct 30 20:58:51.692714 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.692696 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ad5871-dc94-4512-8f9b-3a754d439dca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cacd0ea-b68s2\" (UID: \"54ad5871-dc94-4512-8f9b-3a754d439dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" Oct 30 20:58:51.692829 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.692810 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/499437e4-d0c8-4bb9-972f-9843b0233962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1781e14-pkpww\" (UID: \"499437e4-d0c8-4bb9-972f-9843b0233962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww" Oct 30 20:58:51.747807 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.747783 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsgbw\" (UniqueName: \"kubernetes.io/projected/54ad5871-dc94-4512-8f9b-3a754d439dca-kube-api-access-zsgbw\") pod \"maas-api-revoke-sub-1cacd0ea-b68s2\" (UID: \"54ad5871-dc94-4512-8f9b-3a754d439dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" Oct 30 20:58:51.747906 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.747816 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9mwq\" (UniqueName: \"kubernetes.io/projected/499437e4-d0c8-4bb9-972f-9843b0233962-kube-api-access-d9mwq\") pod \"maas-api-revoke-sub-b1781e14-pkpww\" (UID: \"499437e4-d0c8-4bb9-972f-9843b0233962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww" Oct 30 20:58:51.823053 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.822973 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww" Oct 30 20:58:51.880611 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:51.880586 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" Oct 30 20:58:52.168096 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:52.168009 5755 generic.go:356] "Generic (PLEG): container finished" podID="ec2d186b-4be6-4e85-9614-495004fa6ab2" containerID="e0377bf7cf4e9ca612b6cb3f267411a3ec77e38ba6827113ae2676c0dbf19ae1" exitCode=0 Oct 30 20:58:52.168261 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:52.168091 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj" event={"ID":"ec2d186b-4be6-4e85-9614-495004fa6ab2","Type":"ContainerDied","Data":"e0377bf7cf4e9ca612b6cb3f267411a3ec77e38ba6827113ae2676c0dbf19ae1"} Oct 30 20:58:52.202424 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:52.202378 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww"] Oct 30 20:58:52.202958 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:58:52.202929 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499437e4_d0c8_4bb9_972f_9843b0233962.slice/crio-ed1957c42727822ed751f4a731870f081849d0280210eb2afd6e91786147a903 WatchSource:0}: Error finding container ed1957c42727822ed751f4a731870f081849d0280210eb2afd6e91786147a903: Status 404 returned error can't find the container with id ed1957c42727822ed751f4a731870f081849d0280210eb2afd6e91786147a903 Oct 30 20:58:52.301857 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:52.301834 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2"] Oct 30 20:58:52.302477 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:58:52.302450 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ad5871_dc94_4512_8f9b_3a754d439dca.slice/crio-d0b2b87cc7d8821f345d01c1fb7935632dd362166357c84496ed293a07c58a23 WatchSource:0}: Error finding container d0b2b87cc7d8821f345d01c1fb7935632dd362166357c84496ed293a07c58a23: Status 404 returned error can't find the container with id d0b2b87cc7d8821f345d01c1fb7935632dd362166357c84496ed293a07c58a23 Oct 30 20:58:53.178418 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:53.178348 5755 generic.go:356] "Generic (PLEG): container finished" podID="499437e4-d0c8-4bb9-972f-9843b0233962" containerID="491905521b5debbfa96138d72e8a46a1a14efe2a78d20bf0d201c0fb33cab3ce" exitCode=0 Oct 30 20:58:53.178859 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:53.178439 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww" event={"ID":"499437e4-d0c8-4bb9-972f-9843b0233962","Type":"ContainerDied","Data":"491905521b5debbfa96138d72e8a46a1a14efe2a78d20bf0d201c0fb33cab3ce"} Oct 30 20:58:53.178859 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:53.178494 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww" event={"ID":"499437e4-d0c8-4bb9-972f-9843b0233962","Type":"ContainerStarted","Data":"ed1957c42727822ed751f4a731870f081849d0280210eb2afd6e91786147a903"} Oct 30 20:58:53.180298 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:53.180273 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" event={"ID":"54ad5871-dc94-4512-8f9b-3a754d439dca","Type":"ContainerStarted","Data":"2cd580dec563cf952dca5d8412a7c061e7ffe8acb15f840510a44b58ef17647b"} Oct 30 20:58:53.180446 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:53.180313 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" event={"ID":"54ad5871-dc94-4512-8f9b-3a754d439dca","Type":"ContainerStarted","Data":"d0b2b87cc7d8821f345d01c1fb7935632dd362166357c84496ed293a07c58a23"} Oct 30 20:58:53.338012 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:53.337987 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj" Oct 30 20:58:53.411253 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:53.411184 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec2d186b-4be6-4e85-9614-495004fa6ab2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2d186b-4be6-4e85-9614-495004fa6ab2-maas-api-xks-ca\") pod \"ec2d186b-4be6-4e85-9614-495004fa6ab2\" (UID: \"ec2d186b-4be6-4e85-9614-495004fa6ab2\") " Oct 30 20:58:53.411253 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:53.411218 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2d186b-4be6-4e85-9614-495004fa6ab2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d186b-4be6-4e85-9614-495004fa6ab2-maas-api-openshift-ca\") pod \"ec2d186b-4be6-4e85-9614-495004fa6ab2\" (UID: \"ec2d186b-4be6-4e85-9614-495004fa6ab2\") " Oct 30 20:58:53.411467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:53.411307 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2d186b-4be6-4e85-9614-495004fa6ab2-kube-api-access-mrknn\" (UniqueName: \"kubernetes.io/projected/ec2d186b-4be6-4e85-9614-495004fa6ab2-kube-api-access-mrknn\") pod \"ec2d186b-4be6-4e85-9614-495004fa6ab2\" (UID: \"ec2d186b-4be6-4e85-9614-495004fa6ab2\") " Oct 30 20:58:53.411600 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:53.411578 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2d186b-4be6-4e85-9614-495004fa6ab2-maas-api-openshift-ca" pod "ec2d186b-4be6-4e85-9614-495004fa6ab2" (UID: "ec2d186b-4be6-4e85-9614-495004fa6ab2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:53.411760 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:53.411742 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d186b-4be6-4e85-9614-495004fa6ab2-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:53.413478 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:53.413452 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2d186b-4be6-4e85-9614-495004fa6ab2-maas-api-xks-ca" pod "ec2d186b-4be6-4e85-9614-495004fa6ab2" (UID: "ec2d186b-4be6-4e85-9614-495004fa6ab2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:53.413568 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:53.413514 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2d186b-4be6-4e85-9614-495004fa6ab2-kube-api-access-mrknn" pod "ec2d186b-4be6-4e85-9614-495004fa6ab2" (UID: "ec2d186b-4be6-4e85-9614-495004fa6ab2"). InnerVolumeSpecName "kube-api-access-mrknn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:53.513249 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:53.513209 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2d186b-4be6-4e85-9614-495004fa6ab2-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:53.513249 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:53.513242 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrknn\" (UniqueName: \"kubernetes.io/projected/ec2d186b-4be6-4e85-9614-495004fa6ab2-kube-api-access-mrknn\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.188700 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.188671 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj" Oct 30 20:58:54.189143 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.188671 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj" event={"ID":"ec2d186b-4be6-4e85-9614-495004fa6ab2","Type":"ContainerDied","Data":"8149c7a7bc7586df0bb138bd1ce582a85e358bd5e1abde7acb789670095f7c5f"} Oct 30 20:58:54.189143 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.188800 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8149c7a7bc7586df0bb138bd1ce582a85e358bd5e1abde7acb789670095f7c5f" Oct 30 20:58:54.190358 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.190330 5755 generic.go:356] "Generic (PLEG): container finished" podID="54ad5871-dc94-4512-8f9b-3a754d439dca" containerID="2cd580dec563cf952dca5d8412a7c061e7ffe8acb15f840510a44b58ef17647b" exitCode=0 Oct 30 20:58:54.190486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.190407 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" event={"ID":"54ad5871-dc94-4512-8f9b-3a754d439dca","Type":"ContainerDied","Data":"2cd580dec563cf952dca5d8412a7c061e7ffe8acb15f840510a44b58ef17647b"} Oct 30 20:58:54.371007 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.370984 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww" Oct 30 20:58:54.374512 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.374491 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" Oct 30 20:58:54.425427 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.425370 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/499437e4-d0c8-4bb9-972f-9843b0233962-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/499437e4-d0c8-4bb9-972f-9843b0233962-maas-api-openshift-ca\") pod \"499437e4-d0c8-4bb9-972f-9843b0233962\" (UID: \"499437e4-d0c8-4bb9-972f-9843b0233962\") " Oct 30 20:58:54.425626 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.425606 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/499437e4-d0c8-4bb9-972f-9843b0233962-kube-api-access-d9mwq\" (UniqueName: \"kubernetes.io/projected/499437e4-d0c8-4bb9-972f-9843b0233962-kube-api-access-d9mwq\") pod \"499437e4-d0c8-4bb9-972f-9843b0233962\" (UID: \"499437e4-d0c8-4bb9-972f-9843b0233962\") " Oct 30 20:58:54.425744 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.425725 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/499437e4-d0c8-4bb9-972f-9843b0233962-maas-api-openshift-ca" pod "499437e4-d0c8-4bb9-972f-9843b0233962" (UID: "499437e4-d0c8-4bb9-972f-9843b0233962"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:54.425807 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.425737 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ad5871-dc94-4512-8f9b-3a754d439dca-kube-api-access-zsgbw\" (UniqueName: \"kubernetes.io/projected/54ad5871-dc94-4512-8f9b-3a754d439dca-kube-api-access-zsgbw\") pod \"54ad5871-dc94-4512-8f9b-3a754d439dca\" (UID: \"54ad5871-dc94-4512-8f9b-3a754d439dca\") " Oct 30 20:58:54.425807 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.425782 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ad5871-dc94-4512-8f9b-3a754d439dca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ad5871-dc94-4512-8f9b-3a754d439dca-maas-api-openshift-ca\") pod \"54ad5871-dc94-4512-8f9b-3a754d439dca\" (UID: \"54ad5871-dc94-4512-8f9b-3a754d439dca\") " Oct 30 20:58:54.425889 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.425822 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/499437e4-d0c8-4bb9-972f-9843b0233962-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/499437e4-d0c8-4bb9-972f-9843b0233962-maas-api-xks-ca\") pod \"499437e4-d0c8-4bb9-972f-9843b0233962\" (UID: \"499437e4-d0c8-4bb9-972f-9843b0233962\") " Oct 30 20:58:54.425889 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.425855 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ad5871-dc94-4512-8f9b-3a754d439dca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ad5871-dc94-4512-8f9b-3a754d439dca-maas-api-xks-ca\") pod \"54ad5871-dc94-4512-8f9b-3a754d439dca\" (UID: \"54ad5871-dc94-4512-8f9b-3a754d439dca\") " Oct 30 20:58:54.426119 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.426095 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ad5871-dc94-4512-8f9b-3a754d439dca-maas-api-openshift-ca" pod "54ad5871-dc94-4512-8f9b-3a754d439dca" (UID: "54ad5871-dc94-4512-8f9b-3a754d439dca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:54.426462 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.426441 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ad5871-dc94-4512-8f9b-3a754d439dca-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.426563 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.426465 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/499437e4-d0c8-4bb9-972f-9843b0233962-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.428133 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.428101 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/499437e4-d0c8-4bb9-972f-9843b0233962-maas-api-xks-ca" pod "499437e4-d0c8-4bb9-972f-9843b0233962" (UID: "499437e4-d0c8-4bb9-972f-9843b0233962"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:54.428467 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.428448 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ad5871-dc94-4512-8f9b-3a754d439dca-kube-api-access-zsgbw" pod "54ad5871-dc94-4512-8f9b-3a754d439dca" (UID: "54ad5871-dc94-4512-8f9b-3a754d439dca"). InnerVolumeSpecName "kube-api-access-zsgbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:54.428669 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.428641 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499437e4-d0c8-4bb9-972f-9843b0233962-kube-api-access-d9mwq" pod "499437e4-d0c8-4bb9-972f-9843b0233962" (UID: "499437e4-d0c8-4bb9-972f-9843b0233962"). InnerVolumeSpecName "kube-api-access-d9mwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:54.428780 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.428641 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ad5871-dc94-4512-8f9b-3a754d439dca-maas-api-xks-ca" pod "54ad5871-dc94-4512-8f9b-3a754d439dca" (UID: "54ad5871-dc94-4512-8f9b-3a754d439dca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:54.527798 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.527765 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9mwq\" (UniqueName: \"kubernetes.io/projected/499437e4-d0c8-4bb9-972f-9843b0233962-kube-api-access-d9mwq\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.527798 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.527794 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsgbw\" (UniqueName: \"kubernetes.io/projected/54ad5871-dc94-4512-8f9b-3a754d439dca-kube-api-access-zsgbw\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.527798 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.527806 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/499437e4-d0c8-4bb9-972f-9843b0233962-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.528021 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.527814 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ad5871-dc94-4512-8f9b-3a754d439dca-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.629931 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.629897 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n"] Oct 30 20:58:54.648749 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:54.648718 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk"] Oct 30 20:58:55.198479 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:55.198447 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww" Oct 30 20:58:55.198908 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:55.198453 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww" event={"ID":"499437e4-d0c8-4bb9-972f-9843b0233962","Type":"ContainerDied","Data":"ed1957c42727822ed751f4a731870f081849d0280210eb2afd6e91786147a903"} Oct 30 20:58:55.198908 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:55.198608 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed1957c42727822ed751f4a731870f081849d0280210eb2afd6e91786147a903" Oct 30 20:58:55.199998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:55.199982 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" Oct 30 20:58:55.200097 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:55.200027 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2" event={"ID":"54ad5871-dc94-4512-8f9b-3a754d439dca","Type":"ContainerDied","Data":"d0b2b87cc7d8821f345d01c1fb7935632dd362166357c84496ed293a07c58a23"} Oct 30 20:58:55.200097 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:58:55.200066 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0b2b87cc7d8821f345d01c1fb7935632dd362166357c84496ed293a07c58a23" Oct 30 20:59:04.463212 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:04.463134 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq"] Oct 30 20:59:04.512916 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:04.512879 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3d7c11-q8htq"] Oct 30 20:59:05.211877 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:05.211845 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cfc7d29-6fed-4ef9-be4f-b30d6a59c069" path="/var/lib/kubelet/pods/8cfc7d29-6fed-4ef9-be4f-b30d6a59c069/volumes" Oct 30 20:59:07.454962 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:07.454922 5755 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal" podUID="b1b5f93e01c03391b49a05418cff0a8e" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:09.651229 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:09.651185 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" containerID="cri-o://d999260c4cdcb302b6a30f0d7e2c0bcaa7e5b8435e8d7988fb3ed90fd63d03ec" gracePeriod=45 Oct 30 20:59:09.669849 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:09.669795 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" containerID="cri-o://cc55acd2c3f0a5f7001fb183751c9ddba9459921c04bdb14bb906175700011d5" gracePeriod=45 Oct 30 20:59:09.966859 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:09.966838 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:59:09.970548 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:09.970524 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:59:10.125887 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.125855 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-tls-certs\" (UniqueName: \"kubernetes.io/secret/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-tls-certs\") pod \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " Oct 30 20:59:10.126074 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.125902 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-kube-api-access-tszx2\" (UniqueName: \"kubernetes.io/projected/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-kube-api-access-tszx2\") pod \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " Oct 30 20:59:10.126074 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.125932 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8dce353-15d0-4501-9315-bdf52c45f3c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8dce353-15d0-4501-9315-bdf52c45f3c5-tls-certs\") pod \"f8dce353-15d0-4501-9315-bdf52c45f3c5\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " Oct 30 20:59:10.126074 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.125991 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-tmp-dir\") pod \"f8dce353-15d0-4501-9315-bdf52c45f3c5\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " Oct 30 20:59:10.126296 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.126018 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-home\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-home\") pod \"f8dce353-15d0-4501-9315-bdf52c45f3c5\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " Oct 30 20:59:10.126462 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.126368 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-model-cache\") pod \"f8dce353-15d0-4501-9315-bdf52c45f3c5\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " Oct 30 20:59:10.126551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.126491 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8dce353-15d0-4501-9315-bdf52c45f3c5-kube-api-access-28mm6\" (UniqueName: \"kubernetes.io/projected/f8dce353-15d0-4501-9315-bdf52c45f3c5-kube-api-access-28mm6\") pod \"f8dce353-15d0-4501-9315-bdf52c45f3c5\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " Oct 30 20:59:10.126551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.126530 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-home\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-home\") pod \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " Oct 30 20:59:10.126670 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.126572 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-tmp-dir" pod "f8dce353-15d0-4501-9315-bdf52c45f3c5" (UID: "f8dce353-15d0-4501-9315-bdf52c45f3c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:10.126670 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.126579 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-tmp-dir\") pod \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " Oct 30 20:59:10.126670 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.126645 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-dshm\") pod \"f8dce353-15d0-4501-9315-bdf52c45f3c5\" (UID: \"f8dce353-15d0-4501-9315-bdf52c45f3c5\") " Oct 30 20:59:10.126834 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.126716 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-model-cache\") pod \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " Oct 30 20:59:10.126834 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.126740 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-model-cache" pod "f8dce353-15d0-4501-9315-bdf52c45f3c5" (UID: "f8dce353-15d0-4501-9315-bdf52c45f3c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:10.126834 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.126794 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-dshm\") pod \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\" (UID: \"99b3c6d1-1ca9-4102-a550-2c0cb353f18c\") " Oct 30 20:59:10.126990 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.126883 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-home" pod "f8dce353-15d0-4501-9315-bdf52c45f3c5" (UID: "f8dce353-15d0-4501-9315-bdf52c45f3c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:10.127056 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.127030 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-tmp-dir" pod "99b3c6d1-1ca9-4102-a550-2c0cb353f18c" (UID: "99b3c6d1-1ca9-4102-a550-2c0cb353f18c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:10.127729 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.127413 5755 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-tmp-dir\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:10.127729 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.127437 5755 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-home\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:10.127729 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.127448 5755 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-model-cache\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:10.127729 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.127460 5755 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-tmp-dir\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:10.127729 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.127679 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-model-cache" pod "99b3c6d1-1ca9-4102-a550-2c0cb353f18c" (UID: "99b3c6d1-1ca9-4102-a550-2c0cb353f18c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:10.128058 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.127824 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-home" pod "99b3c6d1-1ca9-4102-a550-2c0cb353f18c" (UID: "99b3c6d1-1ca9-4102-a550-2c0cb353f18c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:10.129249 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.129223 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-tls-certs" pod "99b3c6d1-1ca9-4102-a550-2c0cb353f18c" (UID: "99b3c6d1-1ca9-4102-a550-2c0cb353f18c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:10.129437 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.129413 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8dce353-15d0-4501-9315-bdf52c45f3c5-tls-certs" pod "f8dce353-15d0-4501-9315-bdf52c45f3c5" (UID: "f8dce353-15d0-4501-9315-bdf52c45f3c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:10.129841 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.129816 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-dshm" pod "99b3c6d1-1ca9-4102-a550-2c0cb353f18c" (UID: "99b3c6d1-1ca9-4102-a550-2c0cb353f18c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:10.130167 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.130145 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-kube-api-access-tszx2" pod "99b3c6d1-1ca9-4102-a550-2c0cb353f18c" (UID: "99b3c6d1-1ca9-4102-a550-2c0cb353f18c"). InnerVolumeSpecName "kube-api-access-tszx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:10.130716 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.130696 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-dshm" pod "f8dce353-15d0-4501-9315-bdf52c45f3c5" (UID: "f8dce353-15d0-4501-9315-bdf52c45f3c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:10.130793 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.130740 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8dce353-15d0-4501-9315-bdf52c45f3c5-kube-api-access-28mm6" pod "f8dce353-15d0-4501-9315-bdf52c45f3c5" (UID: "f8dce353-15d0-4501-9315-bdf52c45f3c5"). InnerVolumeSpecName "kube-api-access-28mm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:10.228250 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.228220 5755 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-tls-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:10.228250 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.228248 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tszx2\" (UniqueName: \"kubernetes.io/projected/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-kube-api-access-tszx2\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:10.228457 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.228259 5755 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8dce353-15d0-4501-9315-bdf52c45f3c5-tls-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:10.228457 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.228268 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28mm6\" (UniqueName: \"kubernetes.io/projected/f8dce353-15d0-4501-9315-bdf52c45f3c5-kube-api-access-28mm6\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:10.228457 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.228276 5755 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-home\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:10.228457 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.228285 5755 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8dce353-15d0-4501-9315-bdf52c45f3c5-dshm\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:10.228457 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.228292 5755 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-model-cache\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:10.228457 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.228300 5755 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b3c6d1-1ca9-4102-a550-2c0cb353f18c-dshm\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:10.305179 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.305143 5755 generic.go:356] "Generic (PLEG): container finished" podID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerID="d999260c4cdcb302b6a30f0d7e2c0bcaa7e5b8435e8d7988fb3ed90fd63d03ec" exitCode=0 Oct 30 20:59:10.305350 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.305189 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" Oct 30 20:59:10.305350 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.305192 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" event={"ID":"f8dce353-15d0-4501-9315-bdf52c45f3c5","Type":"ContainerDied","Data":"d999260c4cdcb302b6a30f0d7e2c0bcaa7e5b8435e8d7988fb3ed90fd63d03ec"} Oct 30 20:59:10.305350 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.305236 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n" event={"ID":"f8dce353-15d0-4501-9315-bdf52c45f3c5","Type":"ContainerDied","Data":"2e639f6ac6d3179a38e7c5dec1e17c43b1bab42a683ebf542f9b2744bcd6b57b"} Oct 30 20:59:10.305350 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.305257 5755 scope.go:122] "RemoveContainer" containerID="d999260c4cdcb302b6a30f0d7e2c0bcaa7e5b8435e8d7988fb3ed90fd63d03ec" Oct 30 20:59:10.306792 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.306772 5755 generic.go:356] "Generic (PLEG): container finished" podID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerID="cc55acd2c3f0a5f7001fb183751c9ddba9459921c04bdb14bb906175700011d5" exitCode=0 Oct 30 20:59:10.306928 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.306832 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" event={"ID":"99b3c6d1-1ca9-4102-a550-2c0cb353f18c","Type":"ContainerDied","Data":"cc55acd2c3f0a5f7001fb183751c9ddba9459921c04bdb14bb906175700011d5"} Oct 30 20:59:10.306928 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.306859 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" event={"ID":"99b3c6d1-1ca9-4102-a550-2c0cb353f18c","Type":"ContainerDied","Data":"68128c789f232ae10d559f273ea362bf600f0e1e2f6cd922f7f895ef55845fbc"} Oct 30 20:59:10.306928 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.306877 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk" Oct 30 20:59:10.320729 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.320711 5755 scope.go:122] "RemoveContainer" containerID="d999260c4cdcb302b6a30f0d7e2c0bcaa7e5b8435e8d7988fb3ed90fd63d03ec" Oct 30 20:59:10.321082 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:59:10.321052 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d999260c4cdcb302b6a30f0d7e2c0bcaa7e5b8435e8d7988fb3ed90fd63d03ec\": container with ID starting with d999260c4cdcb302b6a30f0d7e2c0bcaa7e5b8435e8d7988fb3ed90fd63d03ec not found: ID does not exist" containerID="d999260c4cdcb302b6a30f0d7e2c0bcaa7e5b8435e8d7988fb3ed90fd63d03ec" Oct 30 20:59:10.321142 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.321093 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d999260c4cdcb302b6a30f0d7e2c0bcaa7e5b8435e8d7988fb3ed90fd63d03ec"} err="failed to get container status \"d999260c4cdcb302b6a30f0d7e2c0bcaa7e5b8435e8d7988fb3ed90fd63d03ec\": rpc error: code = NotFound desc = could not find container \"d999260c4cdcb302b6a30f0d7e2c0bcaa7e5b8435e8d7988fb3ed90fd63d03ec\": container with ID starting with d999260c4cdcb302b6a30f0d7e2c0bcaa7e5b8435e8d7988fb3ed90fd63d03ec not found: ID does not exist" Oct 30 20:59:10.321142 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.321116 5755 scope.go:122] "RemoveContainer" containerID="cc55acd2c3f0a5f7001fb183751c9ddba9459921c04bdb14bb906175700011d5" Oct 30 20:59:10.330126 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.330109 5755 scope.go:122] "RemoveContainer" containerID="cc55acd2c3f0a5f7001fb183751c9ddba9459921c04bdb14bb906175700011d5" Oct 30 20:59:10.330484 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:59:10.330449 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc55acd2c3f0a5f7001fb183751c9ddba9459921c04bdb14bb906175700011d5\": container with ID starting with cc55acd2c3f0a5f7001fb183751c9ddba9459921c04bdb14bb906175700011d5 not found: ID does not exist" containerID="cc55acd2c3f0a5f7001fb183751c9ddba9459921c04bdb14bb906175700011d5" Oct 30 20:59:10.330596 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.330493 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc55acd2c3f0a5f7001fb183751c9ddba9459921c04bdb14bb906175700011d5"} err="failed to get container status \"cc55acd2c3f0a5f7001fb183751c9ddba9459921c04bdb14bb906175700011d5\": rpc error: code = NotFound desc = could not find container \"cc55acd2c3f0a5f7001fb183751c9ddba9459921c04bdb14bb906175700011d5\": container with ID starting with cc55acd2c3f0a5f7001fb183751c9ddba9459921c04bdb14bb906175700011d5 not found: ID does not exist" Oct 30 20:59:10.473115 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.473082 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n"] Oct 30 20:59:10.478300 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.478238 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-386743/distinct-w5-386743-kserve-6dffd6bf4f-p5g9n"] Oct 30 20:59:10.588842 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.588808 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk"] Oct 30 20:59:10.591644 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:10.591621 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-386743/simulator-w5-386743-kserve-55747d678c-lkcjk"] Oct 30 20:59:11.211871 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:11.211836 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" path="/var/lib/kubelet/pods/99b3c6d1-1ca9-4102-a550-2c0cb353f18c/volumes" Oct 30 20:59:11.212289 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:11.212274 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" path="/var/lib/kubelet/pods/f8dce353-15d0-4501-9315-bdf52c45f3c5/volumes" Oct 30 20:59:11.255670 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:59:11.255637 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:59:13.403277 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.403250 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv"] Oct 30 20:59:13.403798 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.403783 5755 cpu_manager.go:414] "containerMap: removing container" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" Oct 30 20:59:13.403842 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.403799 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" Oct 30 20:59:13.403842 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.403813 5755 cpu_manager.go:414] "containerMap: removing container" podUID="499437e4-d0c8-4bb9-972f-9843b0233962" containerName="revoke-keys" Oct 30 20:59:13.403842 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.403821 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="499437e4-d0c8-4bb9-972f-9843b0233962" containerName="revoke-keys" Oct 30 20:59:13.403842 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.403831 5755 cpu_manager.go:414] "containerMap: removing container" podUID="54ad5871-dc94-4512-8f9b-3a754d439dca" containerName="revoke-keys" Oct 30 20:59:13.403842 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.403836 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ad5871-dc94-4512-8f9b-3a754d439dca" containerName="revoke-keys" Oct 30 20:59:13.403993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.403845 5755 cpu_manager.go:414] "containerMap: removing container" podUID="ec2d186b-4be6-4e85-9614-495004fa6ab2" containerName="revoke-keys" Oct 30 20:59:13.403993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.403850 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2d186b-4be6-4e85-9614-495004fa6ab2" containerName="revoke-keys" Oct 30 20:59:13.403993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.403867 5755 cpu_manager.go:414] "containerMap: removing container" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" Oct 30 20:59:13.403993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.403872 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" Oct 30 20:59:13.403993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.403959 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="99b3c6d1-1ca9-4102-a550-2c0cb353f18c" containerName="main" Oct 30 20:59:13.403993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.403967 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="499437e4-d0c8-4bb9-972f-9843b0233962" containerName="revoke-keys" Oct 30 20:59:13.403993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.403974 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ad5871-dc94-4512-8f9b-3a754d439dca" containerName="revoke-keys" Oct 30 20:59:13.403993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.403982 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2d186b-4be6-4e85-9614-495004fa6ab2" containerName="revoke-keys" Oct 30 20:59:13.403993 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.403989 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8dce353-15d0-4501-9315-bdf52c45f3c5" containerName="main" Oct 30 20:59:13.408829 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.408809 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv" Oct 30 20:59:13.466559 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.466526 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv"] Oct 30 20:59:13.512110 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.512080 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:59:13.512272 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.512253 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:59:13.568974 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.568945 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/214c6ae5-0dfe-48b0-b871-403167878a51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv\" (UID: \"214c6ae5-0dfe-48b0-b871-403167878a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv" Oct 30 20:59:13.569179 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.569011 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w57xw\" (UniqueName: \"kubernetes.io/projected/214c6ae5-0dfe-48b0-b871-403167878a51-kube-api-access-w57xw\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv\" (UID: \"214c6ae5-0dfe-48b0-b871-403167878a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv" Oct 30 20:59:13.569254 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.569218 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/214c6ae5-0dfe-48b0-b871-403167878a51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv\" (UID: \"214c6ae5-0dfe-48b0-b871-403167878a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv" Oct 30 20:59:13.671001 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.670904 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/214c6ae5-0dfe-48b0-b871-403167878a51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv\" (UID: \"214c6ae5-0dfe-48b0-b871-403167878a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv" Oct 30 20:59:13.671169 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.671003 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w57xw\" (UniqueName: \"kubernetes.io/projected/214c6ae5-0dfe-48b0-b871-403167878a51-kube-api-access-w57xw\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv\" (UID: \"214c6ae5-0dfe-48b0-b871-403167878a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv" Oct 30 20:59:13.671169 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.671109 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/214c6ae5-0dfe-48b0-b871-403167878a51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv\" (UID: \"214c6ae5-0dfe-48b0-b871-403167878a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv" Oct 30 20:59:13.671550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.671533 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/214c6ae5-0dfe-48b0-b871-403167878a51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv\" (UID: \"214c6ae5-0dfe-48b0-b871-403167878a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv" Oct 30 20:59:13.673610 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.673592 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/214c6ae5-0dfe-48b0-b871-403167878a51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv\" (UID: \"214c6ae5-0dfe-48b0-b871-403167878a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv" Oct 30 20:59:13.729285 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.729248 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w57xw\" (UniqueName: \"kubernetes.io/projected/214c6ae5-0dfe-48b0-b871-403167878a51-kube-api-access-w57xw\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv\" (UID: \"214c6ae5-0dfe-48b0-b871-403167878a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv" Oct 30 20:59:13.834898 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.834866 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp"] Oct 30 20:59:13.838911 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.838893 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" Oct 30 20:59:13.897163 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.897132 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp"] Oct 30 20:59:13.974609 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.974582 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp\" (UID: \"a5601e3e-ded9-4692-8a4f-0ff0335e8c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" Oct 30 20:59:13.974786 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.974768 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp\" (UID: \"a5601e3e-ded9-4692-8a4f-0ff0335e8c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" Oct 30 20:59:13.974831 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:13.974807 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7b2m\" (UniqueName: \"kubernetes.io/projected/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-kube-api-access-k7b2m\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp\" (UID: \"a5601e3e-ded9-4692-8a4f-0ff0335e8c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" Oct 30 20:59:14.022251 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:14.022223 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv" Oct 30 20:59:14.077674 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:14.075902 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp\" (UID: \"a5601e3e-ded9-4692-8a4f-0ff0335e8c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" Oct 30 20:59:14.077674 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:14.076127 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp\" (UID: \"a5601e3e-ded9-4692-8a4f-0ff0335e8c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" Oct 30 20:59:14.077674 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:14.076173 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7b2m\" (UniqueName: \"kubernetes.io/projected/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-kube-api-access-k7b2m\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp\" (UID: \"a5601e3e-ded9-4692-8a4f-0ff0335e8c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" Oct 30 20:59:14.077674 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:14.077331 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp\" (UID: \"a5601e3e-ded9-4692-8a4f-0ff0335e8c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" Oct 30 20:59:14.080051 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:14.079997 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp\" (UID: \"a5601e3e-ded9-4692-8a4f-0ff0335e8c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" Oct 30 20:59:14.134808 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:14.134779 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7b2m\" (UniqueName: \"kubernetes.io/projected/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-kube-api-access-k7b2m\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp\" (UID: \"a5601e3e-ded9-4692-8a4f-0ff0335e8c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" Oct 30 20:59:14.152329 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:14.152301 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" Oct 30 20:59:14.401620 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:14.401599 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv"] Oct 30 20:59:14.402576 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:59:14.402552 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214c6ae5_0dfe_48b0_b871_403167878a51.slice/crio-e71640a8ce1790fa04d43caff3801d71960bdfb53415797896d9e160e36ed42f WatchSource:0}: Error finding container e71640a8ce1790fa04d43caff3801d71960bdfb53415797896d9e160e36ed42f: Status 404 returned error can't find the container with id e71640a8ce1790fa04d43caff3801d71960bdfb53415797896d9e160e36ed42f Oct 30 20:59:14.532129 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:14.532094 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp"] Oct 30 20:59:14.532867 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:59:14.532836 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5601e3e_ded9_4692_8a4f_0ff0335e8c98.slice/crio-637cde2074f0ee63be4c56e21b270195b4d81c4264b19f9e2ea56450b6c6f0ca WatchSource:0}: Error finding container 637cde2074f0ee63be4c56e21b270195b4d81c4264b19f9e2ea56450b6c6f0ca: Status 404 returned error can't find the container with id 637cde2074f0ee63be4c56e21b270195b4d81c4264b19f9e2ea56450b6c6f0ca Oct 30 20:59:15.342989 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:15.342948 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" event={"ID":"a5601e3e-ded9-4692-8a4f-0ff0335e8c98","Type":"ContainerStarted","Data":"d3548f27e5904ff70b18d55d868f95a0b714b82fab4c4012bacd9c59802ceaa1"} Oct 30 20:59:15.343173 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:15.343013 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" event={"ID":"a5601e3e-ded9-4692-8a4f-0ff0335e8c98","Type":"ContainerStarted","Data":"637cde2074f0ee63be4c56e21b270195b4d81c4264b19f9e2ea56450b6c6f0ca"} Oct 30 20:59:15.344812 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:15.344783 5755 generic.go:356] "Generic (PLEG): container finished" podID="214c6ae5-0dfe-48b0-b871-403167878a51" containerID="b739334d4879597f4696d52197992845113a9592a8cb361a1f0bdd13ac61d848" exitCode=0 Oct 30 20:59:15.344934 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:15.344864 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv" event={"ID":"214c6ae5-0dfe-48b0-b871-403167878a51","Type":"ContainerDied","Data":"b739334d4879597f4696d52197992845113a9592a8cb361a1f0bdd13ac61d848"} Oct 30 20:59:15.344934 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:15.344916 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv" event={"ID":"214c6ae5-0dfe-48b0-b871-403167878a51","Type":"ContainerStarted","Data":"e71640a8ce1790fa04d43caff3801d71960bdfb53415797896d9e160e36ed42f"} Oct 30 20:59:15.482159 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:15.482099 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" podStartSLOduration=2.482085993 podStartE2EDuration="2.482085993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:15.477846039 +0000 UTC m=+1918.404402976" watchObservedRunningTime="2026-10-30 20:59:15.482085993 +0000 UTC m=+1918.408642908" Oct 30 20:59:16.355990 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:16.355957 5755 generic.go:356] "Generic (PLEG): container finished" podID="a5601e3e-ded9-4692-8a4f-0ff0335e8c98" containerID="d3548f27e5904ff70b18d55d868f95a0b714b82fab4c4012bacd9c59802ceaa1" exitCode=0 Oct 30 20:59:16.356421 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:16.356129 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" event={"ID":"a5601e3e-ded9-4692-8a4f-0ff0335e8c98","Type":"ContainerDied","Data":"d3548f27e5904ff70b18d55d868f95a0b714b82fab4c4012bacd9c59802ceaa1"} Oct 30 20:59:16.568947 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:16.568919 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv" Oct 30 20:59:16.611816 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:16.611745 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/214c6ae5-0dfe-48b0-b871-403167878a51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/214c6ae5-0dfe-48b0-b871-403167878a51-maas-api-xks-ca\") pod \"214c6ae5-0dfe-48b0-b871-403167878a51\" (UID: \"214c6ae5-0dfe-48b0-b871-403167878a51\") " Oct 30 20:59:16.611816 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:16.611810 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/214c6ae5-0dfe-48b0-b871-403167878a51-kube-api-access-w57xw\" (UniqueName: \"kubernetes.io/projected/214c6ae5-0dfe-48b0-b871-403167878a51-kube-api-access-w57xw\") pod \"214c6ae5-0dfe-48b0-b871-403167878a51\" (UID: \"214c6ae5-0dfe-48b0-b871-403167878a51\") " Oct 30 20:59:16.612027 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:16.611907 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/214c6ae5-0dfe-48b0-b871-403167878a51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/214c6ae5-0dfe-48b0-b871-403167878a51-maas-api-openshift-ca\") pod \"214c6ae5-0dfe-48b0-b871-403167878a51\" (UID: \"214c6ae5-0dfe-48b0-b871-403167878a51\") " Oct 30 20:59:16.612267 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:16.612242 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/214c6ae5-0dfe-48b0-b871-403167878a51-maas-api-openshift-ca" pod "214c6ae5-0dfe-48b0-b871-403167878a51" (UID: "214c6ae5-0dfe-48b0-b871-403167878a51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:16.613988 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:16.613966 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/214c6ae5-0dfe-48b0-b871-403167878a51-maas-api-xks-ca" pod "214c6ae5-0dfe-48b0-b871-403167878a51" (UID: "214c6ae5-0dfe-48b0-b871-403167878a51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:16.614063 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:16.614000 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/214c6ae5-0dfe-48b0-b871-403167878a51-kube-api-access-w57xw" pod "214c6ae5-0dfe-48b0-b871-403167878a51" (UID: "214c6ae5-0dfe-48b0-b871-403167878a51"). InnerVolumeSpecName "kube-api-access-w57xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:16.713312 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:16.713277 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w57xw\" (UniqueName: \"kubernetes.io/projected/214c6ae5-0dfe-48b0-b871-403167878a51-kube-api-access-w57xw\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:16.713312 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:16.713311 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/214c6ae5-0dfe-48b0-b871-403167878a51-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:16.713615 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:16.713327 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/214c6ae5-0dfe-48b0-b871-403167878a51-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:17.363399 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:17.363354 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv" Oct 30 20:59:17.363751 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:17.363357 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv" event={"ID":"214c6ae5-0dfe-48b0-b871-403167878a51","Type":"ContainerDied","Data":"e71640a8ce1790fa04d43caff3801d71960bdfb53415797896d9e160e36ed42f"} Oct 30 20:59:17.363751 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:17.363502 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e71640a8ce1790fa04d43caff3801d71960bdfb53415797896d9e160e36ed42f" Oct 30 20:59:17.507339 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:17.507313 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" Oct 30 20:59:17.521692 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:17.521668 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-kube-api-access-k7b2m\" (UniqueName: \"kubernetes.io/projected/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-kube-api-access-k7b2m\") pod \"a5601e3e-ded9-4692-8a4f-0ff0335e8c98\" (UID: \"a5601e3e-ded9-4692-8a4f-0ff0335e8c98\") " Oct 30 20:59:17.521833 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:17.521761 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-maas-api-openshift-ca\") pod \"a5601e3e-ded9-4692-8a4f-0ff0335e8c98\" (UID: \"a5601e3e-ded9-4692-8a4f-0ff0335e8c98\") " Oct 30 20:59:17.521975 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:17.521942 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-maas-api-xks-ca\") pod \"a5601e3e-ded9-4692-8a4f-0ff0335e8c98\" (UID: \"a5601e3e-ded9-4692-8a4f-0ff0335e8c98\") " Oct 30 20:59:17.522139 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:17.522109 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-maas-api-openshift-ca" pod "a5601e3e-ded9-4692-8a4f-0ff0335e8c98" (UID: "a5601e3e-ded9-4692-8a4f-0ff0335e8c98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:17.522446 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:17.522429 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:17.524376 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:17.524354 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-kube-api-access-k7b2m" pod "a5601e3e-ded9-4692-8a4f-0ff0335e8c98" (UID: "a5601e3e-ded9-4692-8a4f-0ff0335e8c98"). InnerVolumeSpecName "kube-api-access-k7b2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:17.524376 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:17.524357 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-maas-api-xks-ca" pod "a5601e3e-ded9-4692-8a4f-0ff0335e8c98" (UID: "a5601e3e-ded9-4692-8a4f-0ff0335e8c98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:17.623852 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:17.623817 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:17.623852 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:17.623852 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7b2m\" (UniqueName: \"kubernetes.io/projected/a5601e3e-ded9-4692-8a4f-0ff0335e8c98-kube-api-access-k7b2m\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:18.064156 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:18.064124 5755 scope.go:122] "RemoveContainer" containerID="c9083b269030f952b333116908b90bd6388c92937747f15a504e005113aabbcf" Oct 30 20:59:18.370762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:18.370683 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" Oct 30 20:59:18.370762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:18.370706 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp" event={"ID":"a5601e3e-ded9-4692-8a4f-0ff0335e8c98","Type":"ContainerDied","Data":"637cde2074f0ee63be4c56e21b270195b4d81c4264b19f9e2ea56450b6c6f0ca"} Oct 30 20:59:18.370762 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:18.370760 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="637cde2074f0ee63be4c56e21b270195b4d81c4264b19f9e2ea56450b6c6f0ca" Oct 30 20:59:19.697533 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:19.697493 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb"] Oct 30 20:59:19.698062 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:19.697777 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" podUID="89ccbe51-66d9-4fab-836f-50eaff323f79" containerName="istio-proxy" containerID="cri-o://b177f31361405336a932edfb4d1cf81c86ab52d980e2e59921f6084da07b747d" gracePeriod=30 Oct 30 20:59:20.336636 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:20.336602 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56"] Oct 30 20:59:20.336819 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:20.336795 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" podUID="135653bb-b684-408c-9ee5-7a5930f3ff1d" containerName="istio-proxy" containerID="cri-o://0c81c3796b1d11b3104466973abaff770a700cf2dcdb7ebf2f56a9a2baf06fe0" gracePeriod=30 Oct 30 20:59:20.444631 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:20.444602 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk"] Oct 30 20:59:20.444832 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:20.444811 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" podUID="91aa5e94-167d-44d7-b423-54914f1b2182" containerName="istio-proxy" containerID="cri-o://87a13f6a2204e3b825c87c85dbfa04b25b704823ce3d6f8d2e7e377b7e04e049" gracePeriod=30 Oct 30 20:59:20.450552 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:20.450518 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs"] Oct 30 20:59:20.450784 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:20.450754 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" podUID="6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" containerName="istio-proxy" containerID="cri-o://e05fff2a11f52f494775123efbc0673cfa31a64f45478d9829faf17fe76a0546" gracePeriod=30 Oct 30 20:59:22.971607 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:22.971572 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:59:22.972018 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:22.971637 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" podUID="89ccbe51-66d9-4fab-836f-50eaff323f79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:59:23.052614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:23.052578 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2"] Oct 30 20:59:23.078772 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:23.078738 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383f66e1-rxkm2"] Oct 30 20:59:23.212283 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:23.212251 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac7b7aa-6821-4ea6-bbc7-fd0990b98161" path="/var/lib/kubelet/pods/fac7b7aa-6821-4ea6-bbc7-fd0990b98161/volumes" Oct 30 20:59:24.270969 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:24.270935 5755 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:59:24.271449 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:24.271007 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" podUID="135653bb-b684-408c-9ee5-7a5930f3ff1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:59:24.978260 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:24.978234 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:59:25.012448 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.012420 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-token\" (UniqueName: \"kubernetes.io/projected/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-token\") pod \"89ccbe51-66d9-4fab-836f-50eaff323f79\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " Oct 30 20:59:25.012563 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.012507 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89ccbe51-66d9-4fab-836f-50eaff323f79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89ccbe51-66d9-4fab-836f-50eaff323f79-istiod-ca-cert\") pod \"89ccbe51-66d9-4fab-836f-50eaff323f79\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " Oct 30 20:59:25.012624 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.012601 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-data\") pod \"89ccbe51-66d9-4fab-836f-50eaff323f79\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " Oct 30 20:59:25.012663 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.012650 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-workload-certs\") pod \"89ccbe51-66d9-4fab-836f-50eaff323f79\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " Oct 30 20:59:25.012715 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.012687 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-podinfo\") pod \"89ccbe51-66d9-4fab-836f-50eaff323f79\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " Oct 30 20:59:25.012768 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.012731 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ccbe51-66d9-4fab-836f-50eaff323f79-kube-api-access-2f8c6\" (UniqueName: \"kubernetes.io/projected/89ccbe51-66d9-4fab-836f-50eaff323f79-kube-api-access-2f8c6\") pod \"89ccbe51-66d9-4fab-836f-50eaff323f79\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " Oct 30 20:59:25.012768 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.012757 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-workload-socket\") pod \"89ccbe51-66d9-4fab-836f-50eaff323f79\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " Oct 30 20:59:25.012874 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.012809 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-credential-socket\") pod \"89ccbe51-66d9-4fab-836f-50eaff323f79\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " Oct 30 20:59:25.012874 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.012842 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-envoy\") pod \"89ccbe51-66d9-4fab-836f-50eaff323f79\" (UID: \"89ccbe51-66d9-4fab-836f-50eaff323f79\") " Oct 30 20:59:25.012973 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.012905 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89ccbe51-66d9-4fab-836f-50eaff323f79-istiod-ca-cert" pod "89ccbe51-66d9-4fab-836f-50eaff323f79" (UID: "89ccbe51-66d9-4fab-836f-50eaff323f79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:25.012973 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.012909 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-workload-certs" pod "89ccbe51-66d9-4fab-836f-50eaff323f79" (UID: "89ccbe51-66d9-4fab-836f-50eaff323f79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.012973 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.012925 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-data" pod "89ccbe51-66d9-4fab-836f-50eaff323f79" (UID: "89ccbe51-66d9-4fab-836f-50eaff323f79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.013158 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.013138 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-workload-socket" pod "89ccbe51-66d9-4fab-836f-50eaff323f79" (UID: "89ccbe51-66d9-4fab-836f-50eaff323f79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.013243 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.013216 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-credential-socket" pod "89ccbe51-66d9-4fab-836f-50eaff323f79" (UID: "89ccbe51-66d9-4fab-836f-50eaff323f79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.013366 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.013315 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89ccbe51-66d9-4fab-836f-50eaff323f79-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.013366 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.013336 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.013366 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.013349 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.013366 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.013362 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.013584 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.013375 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.014793 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.014773 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-token" pod "89ccbe51-66d9-4fab-836f-50eaff323f79" (UID: "89ccbe51-66d9-4fab-836f-50eaff323f79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:25.014952 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.014929 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ccbe51-66d9-4fab-836f-50eaff323f79-kube-api-access-2f8c6" pod "89ccbe51-66d9-4fab-836f-50eaff323f79" (UID: "89ccbe51-66d9-4fab-836f-50eaff323f79"). InnerVolumeSpecName "kube-api-access-2f8c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:25.015616 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.015592 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-envoy" pod "89ccbe51-66d9-4fab-836f-50eaff323f79" (UID: "89ccbe51-66d9-4fab-836f-50eaff323f79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.015616 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.015603 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-podinfo" pod "89ccbe51-66d9-4fab-836f-50eaff323f79" (UID: "89ccbe51-66d9-4fab-836f-50eaff323f79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:59:25.114871 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.114834 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.114871 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.114868 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f8c6\" (UniqueName: \"kubernetes.io/projected/89ccbe51-66d9-4fab-836f-50eaff323f79-kube-api-access-2f8c6\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.115084 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.114879 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.115084 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.114889 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89ccbe51-66d9-4fab-836f-50eaff323f79-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.470486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.465114 5755 generic.go:356] "Generic (PLEG): container finished" podID="89ccbe51-66d9-4fab-836f-50eaff323f79" containerID="b177f31361405336a932edfb4d1cf81c86ab52d980e2e59921f6084da07b747d" exitCode=0 Oct 30 20:59:25.470486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.465227 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" event={"ID":"89ccbe51-66d9-4fab-836f-50eaff323f79","Type":"ContainerDied","Data":"b177f31361405336a932edfb4d1cf81c86ab52d980e2e59921f6084da07b747d"} Oct 30 20:59:25.470486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.465279 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" event={"ID":"89ccbe51-66d9-4fab-836f-50eaff323f79","Type":"ContainerDied","Data":"a24abe144fb6ea5a13b440431cc5bf4936f2de1b64c5858ac224da67c283b34e"} Oct 30 20:59:25.470486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.465311 5755 scope.go:122] "RemoveContainer" containerID="b177f31361405336a932edfb4d1cf81c86ab52d980e2e59921f6084da07b747d" Oct 30 20:59:25.470486 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.465500 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb" Oct 30 20:59:25.489149 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.489095 5755 scope.go:122] "RemoveContainer" containerID="b177f31361405336a932edfb4d1cf81c86ab52d980e2e59921f6084da07b747d" Oct 30 20:59:25.489499 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:59:25.489469 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b177f31361405336a932edfb4d1cf81c86ab52d980e2e59921f6084da07b747d\": container with ID starting with b177f31361405336a932edfb4d1cf81c86ab52d980e2e59921f6084da07b747d not found: ID does not exist" containerID="b177f31361405336a932edfb4d1cf81c86ab52d980e2e59921f6084da07b747d" Oct 30 20:59:25.489718 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.489507 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b177f31361405336a932edfb4d1cf81c86ab52d980e2e59921f6084da07b747d"} err="failed to get container status \"b177f31361405336a932edfb4d1cf81c86ab52d980e2e59921f6084da07b747d\": rpc error: code = NotFound desc = could not find container \"b177f31361405336a932edfb4d1cf81c86ab52d980e2e59921f6084da07b747d\": container with ID starting with b177f31361405336a932edfb4d1cf81c86ab52d980e2e59921f6084da07b747d not found: ID does not exist" Oct 30 20:59:25.537963 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:59:25.537930 5755 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c005f9c_fdc8_4ae2_bf34_9ffcdf13b044.slice/crio-e05fff2a11f52f494775123efbc0673cfa31a64f45478d9829faf17fe76a0546.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c005f9c_fdc8_4ae2_bf34_9ffcdf13b044.slice/crio-conmon-e05fff2a11f52f494775123efbc0673cfa31a64f45478d9829faf17fe76a0546.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:59:25.644448 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.643507 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb"] Oct 30 20:59:25.648474 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.648436 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-8wnnb"] Oct 30 20:59:25.660650 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.660624 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:59:25.737396 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.737356 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-credential-socket\") pod \"135653bb-b684-408c-9ee5-7a5930f3ff1d\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " Oct 30 20:59:25.737540 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.737433 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135653bb-b684-408c-9ee5-7a5930f3ff1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135653bb-b684-408c-9ee5-7a5930f3ff1d-istiod-ca-cert\") pod \"135653bb-b684-408c-9ee5-7a5930f3ff1d\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " Oct 30 20:59:25.737540 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.737516 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-podinfo\") pod \"135653bb-b684-408c-9ee5-7a5930f3ff1d\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " Oct 30 20:59:25.737655 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.737555 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-workload-socket\") pod \"135653bb-b684-408c-9ee5-7a5930f3ff1d\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " Oct 30 20:59:25.737655 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.737583 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-envoy\") pod \"135653bb-b684-408c-9ee5-7a5930f3ff1d\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " Oct 30 20:59:25.737655 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.737610 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135653bb-b684-408c-9ee5-7a5930f3ff1d-kube-api-access-h2xl5\" (UniqueName: \"kubernetes.io/projected/135653bb-b684-408c-9ee5-7a5930f3ff1d-kube-api-access-h2xl5\") pod \"135653bb-b684-408c-9ee5-7a5930f3ff1d\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " Oct 30 20:59:25.737818 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.737662 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-credential-socket" pod "135653bb-b684-408c-9ee5-7a5930f3ff1d" (UID: "135653bb-b684-408c-9ee5-7a5930f3ff1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.737818 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.737669 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-data\") pod \"135653bb-b684-408c-9ee5-7a5930f3ff1d\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " Oct 30 20:59:25.737935 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.737843 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-data" pod "135653bb-b684-408c-9ee5-7a5930f3ff1d" (UID: "135653bb-b684-408c-9ee5-7a5930f3ff1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.737935 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.737847 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-token\" (UniqueName: \"kubernetes.io/projected/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-token\") pod \"135653bb-b684-408c-9ee5-7a5930f3ff1d\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " Oct 30 20:59:25.737935 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.737901 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135653bb-b684-408c-9ee5-7a5930f3ff1d-istiod-ca-cert" pod "135653bb-b684-408c-9ee5-7a5930f3ff1d" (UID: "135653bb-b684-408c-9ee5-7a5930f3ff1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:25.738091 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.737937 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-workload-certs\") pod \"135653bb-b684-408c-9ee5-7a5930f3ff1d\" (UID: \"135653bb-b684-408c-9ee5-7a5930f3ff1d\") " Oct 30 20:59:25.738285 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.738264 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-workload-certs" pod "135653bb-b684-408c-9ee5-7a5930f3ff1d" (UID: "135653bb-b684-408c-9ee5-7a5930f3ff1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.738378 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.738354 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-workload-socket" pod "135653bb-b684-408c-9ee5-7a5930f3ff1d" (UID: "135653bb-b684-408c-9ee5-7a5930f3ff1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.738655 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.738552 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.738655 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.738578 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.738655 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.738591 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.738655 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.738606 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135653bb-b684-408c-9ee5-7a5930f3ff1d-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.738655 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.738620 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.740776 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.740744 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135653bb-b684-408c-9ee5-7a5930f3ff1d-kube-api-access-h2xl5" pod "135653bb-b684-408c-9ee5-7a5930f3ff1d" (UID: "135653bb-b684-408c-9ee5-7a5930f3ff1d"). InnerVolumeSpecName "kube-api-access-h2xl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:25.740885 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.740854 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-podinfo" pod "135653bb-b684-408c-9ee5-7a5930f3ff1d" (UID: "135653bb-b684-408c-9ee5-7a5930f3ff1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:59:25.741109 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.741090 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-token" pod "135653bb-b684-408c-9ee5-7a5930f3ff1d" (UID: "135653bb-b684-408c-9ee5-7a5930f3ff1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:25.741279 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.741159 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-envoy" pod "135653bb-b684-408c-9ee5-7a5930f3ff1d" (UID: "135653bb-b684-408c-9ee5-7a5930f3ff1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.764007 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.763986 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:59:25.766017 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.766002 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:59:25.839880 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.839854 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-workload-socket\") pod \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " Oct 30 20:59:25.840043 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.839897 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-credential-socket\") pod \"91aa5e94-167d-44d7-b423-54914f1b2182\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " Oct 30 20:59:25.840043 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.839919 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-data\") pod \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " Oct 30 20:59:25.840043 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.839949 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91aa5e94-167d-44d7-b423-54914f1b2182-istio-token\" (UniqueName: \"kubernetes.io/projected/91aa5e94-167d-44d7-b423-54914f1b2182-istio-token\") pod \"91aa5e94-167d-44d7-b423-54914f1b2182\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " Oct 30 20:59:25.840043 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.839986 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-podinfo\") pod \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " Oct 30 20:59:25.840043 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840029 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-envoy\") pod \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " Oct 30 20:59:25.840330 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840058 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-workload-certs\") pod \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " Oct 30 20:59:25.840330 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840098 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-kube-api-access-l6svz\" (UniqueName: \"kubernetes.io/projected/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-kube-api-access-l6svz\") pod \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " Oct 30 20:59:25.840330 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840146 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-workload-certs\") pod \"91aa5e94-167d-44d7-b423-54914f1b2182\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " Oct 30 20:59:25.840330 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840169 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-workload-socket" pod "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" (UID: "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.840330 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840198 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istiod-ca-cert\") pod \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " Oct 30 20:59:25.840330 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840209 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-credential-socket" pod "91aa5e94-167d-44d7-b423-54914f1b2182" (UID: "91aa5e94-167d-44d7-b423-54914f1b2182"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.840330 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840230 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-token\" (UniqueName: \"kubernetes.io/projected/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-token\") pod \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " Oct 30 20:59:25.840330 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840276 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-data" pod "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" (UID: "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.840330 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840292 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91aa5e94-167d-44d7-b423-54914f1b2182-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aa5e94-167d-44d7-b423-54914f1b2182-istiod-ca-cert\") pod \"91aa5e94-167d-44d7-b423-54914f1b2182\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " Oct 30 20:59:25.840823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840349 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-workload-socket\") pod \"91aa5e94-167d-44d7-b423-54914f1b2182\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " Oct 30 20:59:25.840823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840435 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91aa5e94-167d-44d7-b423-54914f1b2182-kube-api-access-h7jk2\" (UniqueName: \"kubernetes.io/projected/91aa5e94-167d-44d7-b423-54914f1b2182-kube-api-access-h7jk2\") pod \"91aa5e94-167d-44d7-b423-54914f1b2182\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " Oct 30 20:59:25.840823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840472 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-istio-data\") pod \"91aa5e94-167d-44d7-b423-54914f1b2182\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " Oct 30 20:59:25.840823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840501 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-istio-envoy\") pod \"91aa5e94-167d-44d7-b423-54914f1b2182\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " Oct 30 20:59:25.840823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840488 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-workload-certs" pod "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" (UID: "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.840823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840551 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91aa5e94-167d-44d7-b423-54914f1b2182-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aa5e94-167d-44d7-b423-54914f1b2182-istio-podinfo\") pod \"91aa5e94-167d-44d7-b423-54914f1b2182\" (UID: \"91aa5e94-167d-44d7-b423-54914f1b2182\") " Oct 30 20:59:25.840823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840610 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-credential-socket\") pod \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\" (UID: \"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044\") " Oct 30 20:59:25.840823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840700 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-workload-certs" pod "91aa5e94-167d-44d7-b423-54914f1b2182" (UID: "91aa5e94-167d-44d7-b423-54914f1b2182"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.840823 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.840630 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istiod-ca-cert" pod "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" (UID: "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:25.841292 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.841163 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91aa5e94-167d-44d7-b423-54914f1b2182-istiod-ca-cert" pod "91aa5e94-167d-44d7-b423-54914f1b2182" (UID: "91aa5e94-167d-44d7-b423-54914f1b2182"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:25.841353 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.841308 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.841353 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.841329 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.841353 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.841344 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.841550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.841358 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.841550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.841372 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aa5e94-167d-44d7-b423-54914f1b2182-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.841550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.841402 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.841550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.841417 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2xl5\" (UniqueName: \"kubernetes.io/projected/135653bb-b684-408c-9ee5-7a5930f3ff1d-kube-api-access-h2xl5\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.841550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.841433 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.841550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.841453 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.841550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.841467 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.841550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.841481 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135653bb-b684-408c-9ee5-7a5930f3ff1d-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.841550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.841450 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-istio-data" pod "91aa5e94-167d-44d7-b423-54914f1b2182" (UID: "91aa5e94-167d-44d7-b423-54914f1b2182"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.841550 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.841463 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-workload-socket" pod "91aa5e94-167d-44d7-b423-54914f1b2182" (UID: "91aa5e94-167d-44d7-b423-54914f1b2182"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.842312 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.841641 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-credential-socket" pod "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" (UID: "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.843235 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.843208 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-envoy" pod "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" (UID: "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.843360 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.843317 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91aa5e94-167d-44d7-b423-54914f1b2182-istio-token" pod "91aa5e94-167d-44d7-b423-54914f1b2182" (UID: "91aa5e94-167d-44d7-b423-54914f1b2182"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:25.843702 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.843671 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-kube-api-access-l6svz" pod "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" (UID: "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044"). InnerVolumeSpecName "kube-api-access-l6svz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:25.843814 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.843698 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-podinfo" pod "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" (UID: "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:59:25.843899 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.843878 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91aa5e94-167d-44d7-b423-54914f1b2182-istio-podinfo" pod "91aa5e94-167d-44d7-b423-54914f1b2182" (UID: "91aa5e94-167d-44d7-b423-54914f1b2182"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:59:25.844105 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.844088 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91aa5e94-167d-44d7-b423-54914f1b2182-kube-api-access-h7jk2" pod "91aa5e94-167d-44d7-b423-54914f1b2182" (UID: "91aa5e94-167d-44d7-b423-54914f1b2182"). InnerVolumeSpecName "kube-api-access-h7jk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:25.844185 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.844163 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-token" pod "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" (UID: "6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:25.844603 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.844589 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-istio-envoy" pod "91aa5e94-167d-44d7-b423-54914f1b2182" (UID: "91aa5e94-167d-44d7-b423-54914f1b2182"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:25.942667 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.942586 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6svz\" (UniqueName: \"kubernetes.io/projected/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-kube-api-access-l6svz\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.942667 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.942622 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.942667 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.942634 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.942667 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.942642 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7jk2\" (UniqueName: \"kubernetes.io/projected/91aa5e94-167d-44d7-b423-54914f1b2182-kube-api-access-h7jk2\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.942667 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.942653 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.942667 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.942661 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aa5e94-167d-44d7-b423-54914f1b2182-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.942667 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.942669 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aa5e94-167d-44d7-b423-54914f1b2182-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.942990 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.942679 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.942990 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.942688 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91aa5e94-167d-44d7-b423-54914f1b2182-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.942990 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.942698 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:25.942990 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:25.942707 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:26.487887 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.487848 5755 generic.go:356] "Generic (PLEG): container finished" podID="6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" containerID="e05fff2a11f52f494775123efbc0673cfa31a64f45478d9829faf17fe76a0546" exitCode=0 Oct 30 20:59:26.488464 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.487941 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" event={"ID":"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044","Type":"ContainerDied","Data":"e05fff2a11f52f494775123efbc0673cfa31a64f45478d9829faf17fe76a0546"} Oct 30 20:59:26.488464 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.487993 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" event={"ID":"6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044","Type":"ContainerDied","Data":"106f3ccaa3961e65497fdfb071282ab2a20291d6a7b484153d8350e075ccd91c"} Oct 30 20:59:26.488464 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.488021 5755 scope.go:122] "RemoveContainer" containerID="e05fff2a11f52f494775123efbc0673cfa31a64f45478d9829faf17fe76a0546" Oct 30 20:59:26.488464 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.488137 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs" Oct 30 20:59:26.502614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.496285 5755 generic.go:356] "Generic (PLEG): container finished" podID="135653bb-b684-408c-9ee5-7a5930f3ff1d" containerID="0c81c3796b1d11b3104466973abaff770a700cf2dcdb7ebf2f56a9a2baf06fe0" exitCode=0 Oct 30 20:59:26.502614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.496436 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" event={"ID":"135653bb-b684-408c-9ee5-7a5930f3ff1d","Type":"ContainerDied","Data":"0c81c3796b1d11b3104466973abaff770a700cf2dcdb7ebf2f56a9a2baf06fe0"} Oct 30 20:59:26.502614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.496487 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" event={"ID":"135653bb-b684-408c-9ee5-7a5930f3ff1d","Type":"ContainerDied","Data":"43771bebee63ac04de648ff6929d0c6839a7c594ed0143a9cec6a5e79ad306a8"} Oct 30 20:59:26.502614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.496563 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56" Oct 30 20:59:26.508448 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.508414 5755 generic.go:356] "Generic (PLEG): container finished" podID="91aa5e94-167d-44d7-b423-54914f1b2182" containerID="87a13f6a2204e3b825c87c85dbfa04b25b704823ce3d6f8d2e7e377b7e04e049" exitCode=0 Oct 30 20:59:26.508569 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.508505 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" event={"ID":"91aa5e94-167d-44d7-b423-54914f1b2182","Type":"ContainerDied","Data":"87a13f6a2204e3b825c87c85dbfa04b25b704823ce3d6f8d2e7e377b7e04e049"} Oct 30 20:59:26.508569 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.508552 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" event={"ID":"91aa5e94-167d-44d7-b423-54914f1b2182","Type":"ContainerDied","Data":"0756d61c4bd1857ae514412628a2cddccf628cc7b4a260f96a39999fcb14a87d"} Oct 30 20:59:26.508668 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.508626 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk" Oct 30 20:59:26.517636 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.517596 5755 scope.go:122] "RemoveContainer" containerID="e05fff2a11f52f494775123efbc0673cfa31a64f45478d9829faf17fe76a0546" Oct 30 20:59:26.517981 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:59:26.517950 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05fff2a11f52f494775123efbc0673cfa31a64f45478d9829faf17fe76a0546\": container with ID starting with e05fff2a11f52f494775123efbc0673cfa31a64f45478d9829faf17fe76a0546 not found: ID does not exist" containerID="e05fff2a11f52f494775123efbc0673cfa31a64f45478d9829faf17fe76a0546" Oct 30 20:59:26.518081 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.517995 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05fff2a11f52f494775123efbc0673cfa31a64f45478d9829faf17fe76a0546"} err="failed to get container status \"e05fff2a11f52f494775123efbc0673cfa31a64f45478d9829faf17fe76a0546\": rpc error: code = NotFound desc = could not find container \"e05fff2a11f52f494775123efbc0673cfa31a64f45478d9829faf17fe76a0546\": container with ID starting with e05fff2a11f52f494775123efbc0673cfa31a64f45478d9829faf17fe76a0546 not found: ID does not exist" Oct 30 20:59:26.518081 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.518019 5755 scope.go:122] "RemoveContainer" containerID="0c81c3796b1d11b3104466973abaff770a700cf2dcdb7ebf2f56a9a2baf06fe0" Oct 30 20:59:26.529324 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.529300 5755 scope.go:122] "RemoveContainer" containerID="0c81c3796b1d11b3104466973abaff770a700cf2dcdb7ebf2f56a9a2baf06fe0" Oct 30 20:59:26.529821 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:59:26.529793 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c81c3796b1d11b3104466973abaff770a700cf2dcdb7ebf2f56a9a2baf06fe0\": container with ID starting with 0c81c3796b1d11b3104466973abaff770a700cf2dcdb7ebf2f56a9a2baf06fe0 not found: ID does not exist" containerID="0c81c3796b1d11b3104466973abaff770a700cf2dcdb7ebf2f56a9a2baf06fe0" Oct 30 20:59:26.529936 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.529830 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c81c3796b1d11b3104466973abaff770a700cf2dcdb7ebf2f56a9a2baf06fe0"} err="failed to get container status \"0c81c3796b1d11b3104466973abaff770a700cf2dcdb7ebf2f56a9a2baf06fe0\": rpc error: code = NotFound desc = could not find container \"0c81c3796b1d11b3104466973abaff770a700cf2dcdb7ebf2f56a9a2baf06fe0\": container with ID starting with 0c81c3796b1d11b3104466973abaff770a700cf2dcdb7ebf2f56a9a2baf06fe0 not found: ID does not exist" Oct 30 20:59:26.529936 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.529850 5755 scope.go:122] "RemoveContainer" containerID="87a13f6a2204e3b825c87c85dbfa04b25b704823ce3d6f8d2e7e377b7e04e049" Oct 30 20:59:26.539544 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.539523 5755 scope.go:122] "RemoveContainer" containerID="87a13f6a2204e3b825c87c85dbfa04b25b704823ce3d6f8d2e7e377b7e04e049" Oct 30 20:59:26.539855 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:59:26.539826 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a13f6a2204e3b825c87c85dbfa04b25b704823ce3d6f8d2e7e377b7e04e049\": container with ID starting with 87a13f6a2204e3b825c87c85dbfa04b25b704823ce3d6f8d2e7e377b7e04e049 not found: ID does not exist" containerID="87a13f6a2204e3b825c87c85dbfa04b25b704823ce3d6f8d2e7e377b7e04e049" Oct 30 20:59:26.539919 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.539862 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a13f6a2204e3b825c87c85dbfa04b25b704823ce3d6f8d2e7e377b7e04e049"} err="failed to get container status \"87a13f6a2204e3b825c87c85dbfa04b25b704823ce3d6f8d2e7e377b7e04e049\": rpc error: code = NotFound desc = could not find container \"87a13f6a2204e3b825c87c85dbfa04b25b704823ce3d6f8d2e7e377b7e04e049\": container with ID starting with 87a13f6a2204e3b825c87c85dbfa04b25b704823ce3d6f8d2e7e377b7e04e049 not found: ID does not exist" Oct 30 20:59:26.631000 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.630935 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk"] Oct 30 20:59:26.635673 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.635645 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f6ebf5-2nrqk"] Oct 30 20:59:26.658025 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.657992 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56"] Oct 30 20:59:26.663005 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.662982 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-fsf56"] Oct 30 20:59:26.785076 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.784981 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs"] Oct 30 20:59:26.785076 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.785044 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-xdkgs"] Oct 30 20:59:26.894437 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.894406 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk"] Oct 30 20:59:26.900088 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:26.900063 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-rrzvk"] Oct 30 20:59:27.212515 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:27.212445 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135653bb-b684-408c-9ee5-7a5930f3ff1d" path="/var/lib/kubelet/pods/135653bb-b684-408c-9ee5-7a5930f3ff1d/volumes" Oct 30 20:59:27.212892 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:27.212878 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" path="/var/lib/kubelet/pods/6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044/volumes" Oct 30 20:59:27.213265 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:27.213254 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89ccbe51-66d9-4fab-836f-50eaff323f79" path="/var/lib/kubelet/pods/89ccbe51-66d9-4fab-836f-50eaff323f79/volumes" Oct 30 20:59:27.213867 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:27.213850 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91aa5e94-167d-44d7-b423-54914f1b2182" path="/var/lib/kubelet/pods/91aa5e94-167d-44d7-b423-54914f1b2182/volumes" Oct 30 20:59:27.214237 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:27.214224 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f" path="/var/lib/kubelet/pods/dc6b7bdc-b19c-4370-8d36-1d90c4e8e63f/volumes" Oct 30 20:59:33.127492 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:33.127456 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg"] Oct 30 20:59:33.171942 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:33.171910 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b43c84-mjwwg"] Oct 30 20:59:33.211510 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:33.211475 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf3fa41-c39a-4099-ae01-6ae6495a3bf6" path="/var/lib/kubelet/pods/bcf3fa41-c39a-4099-ae01-6ae6495a3bf6/volumes" Oct 30 20:59:35.121570 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:35.121536 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6"] Oct 30 20:59:35.122007 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:35.121755 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" podUID="dec4885f-2622-40b5-bc12-e15f5ec1e51c" containerName="istio-proxy" containerID="cri-o://3a257893ff509f695560ca9a3dbd1820eb46808f2f3ae64ab1dadb8bf6e9fc40" gracePeriod=30 Oct 30 20:59:37.547517 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.547481 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j"] Oct 30 20:59:37.547940 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.547928 5755 cpu_manager.go:414] "containerMap: removing container" podUID="a5601e3e-ded9-4692-8a4f-0ff0335e8c98" containerName="revoke-keys" Oct 30 20:59:37.547998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.547941 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5601e3e-ded9-4692-8a4f-0ff0335e8c98" containerName="revoke-keys" Oct 30 20:59:37.547998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.547965 5755 cpu_manager.go:414] "containerMap: removing container" podUID="214c6ae5-0dfe-48b0-b871-403167878a51" containerName="revoke-keys" Oct 30 20:59:37.547998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.547970 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="214c6ae5-0dfe-48b0-b871-403167878a51" containerName="revoke-keys" Oct 30 20:59:37.547998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.547978 5755 cpu_manager.go:414] "containerMap: removing container" podUID="6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" containerName="istio-proxy" Oct 30 20:59:37.547998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.547983 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" containerName="istio-proxy" Oct 30 20:59:37.547998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.547993 5755 cpu_manager.go:414] "containerMap: removing container" podUID="89ccbe51-66d9-4fab-836f-50eaff323f79" containerName="istio-proxy" Oct 30 20:59:37.547998 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.547998 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ccbe51-66d9-4fab-836f-50eaff323f79" containerName="istio-proxy" Oct 30 20:59:37.548225 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.548005 5755 cpu_manager.go:414] "containerMap: removing container" podUID="91aa5e94-167d-44d7-b423-54914f1b2182" containerName="istio-proxy" Oct 30 20:59:37.548225 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.548009 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91aa5e94-167d-44d7-b423-54914f1b2182" containerName="istio-proxy" Oct 30 20:59:37.548225 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.548018 5755 cpu_manager.go:414] "containerMap: removing container" podUID="135653bb-b684-408c-9ee5-7a5930f3ff1d" containerName="istio-proxy" Oct 30 20:59:37.548225 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.548022 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135653bb-b684-408c-9ee5-7a5930f3ff1d" containerName="istio-proxy" Oct 30 20:59:37.548225 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.548092 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5601e3e-ded9-4692-8a4f-0ff0335e8c98" containerName="revoke-keys" Oct 30 20:59:37.548225 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.548103 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="214c6ae5-0dfe-48b0-b871-403167878a51" containerName="revoke-keys" Oct 30 20:59:37.548225 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.548110 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c005f9c-fdc8-4ae2-bf34-9ffcdf13b044" containerName="istio-proxy" Oct 30 20:59:37.548225 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.548116 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="89ccbe51-66d9-4fab-836f-50eaff323f79" containerName="istio-proxy" Oct 30 20:59:37.548225 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.548122 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="91aa5e94-167d-44d7-b423-54914f1b2182" containerName="istio-proxy" Oct 30 20:59:37.548225 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.548128 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="135653bb-b684-408c-9ee5-7a5930f3ff1d" containerName="istio-proxy" Oct 30 20:59:37.551189 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.551174 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" Oct 30 20:59:37.596053 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.596022 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7djf\" (UniqueName: \"kubernetes.io/projected/f91abf06-f896-4a66-9653-1e7533769692-kube-api-access-d7djf\") pod \"maas-api-revoke-sub-c641beaf-nqd9j\" (UID: \"f91abf06-f896-4a66-9653-1e7533769692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" Oct 30 20:59:37.596209 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.596076 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91abf06-f896-4a66-9653-1e7533769692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c641beaf-nqd9j\" (UID: \"f91abf06-f896-4a66-9653-1e7533769692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" Oct 30 20:59:37.596252 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.596201 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91abf06-f896-4a66-9653-1e7533769692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c641beaf-nqd9j\" (UID: \"f91abf06-f896-4a66-9653-1e7533769692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" Oct 30 20:59:37.600168 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.600144 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k"] Oct 30 20:59:37.603761 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.603746 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" Oct 30 20:59:37.611440 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.611415 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j"] Oct 30 20:59:37.647049 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.647014 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:59:37.647211 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.647074 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 20:59:37.671414 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.671361 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k"] Oct 30 20:59:37.697071 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.697039 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9392c8-52fd-48f2-af13-a7df13651950-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce6f2b33-gq47k\" (UID: \"0e9392c8-52fd-48f2-af13-a7df13651950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" Oct 30 20:59:37.697223 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.697185 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7djf\" (UniqueName: \"kubernetes.io/projected/f91abf06-f896-4a66-9653-1e7533769692-kube-api-access-d7djf\") pod \"maas-api-revoke-sub-c641beaf-nqd9j\" (UID: \"f91abf06-f896-4a66-9653-1e7533769692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" Oct 30 20:59:37.697301 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.697280 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91abf06-f896-4a66-9653-1e7533769692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c641beaf-nqd9j\" (UID: \"f91abf06-f896-4a66-9653-1e7533769692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" Oct 30 20:59:37.697466 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.697446 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91abf06-f896-4a66-9653-1e7533769692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c641beaf-nqd9j\" (UID: \"f91abf06-f896-4a66-9653-1e7533769692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" Oct 30 20:59:37.697551 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.697510 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtr69\" (UniqueName: \"kubernetes.io/projected/0e9392c8-52fd-48f2-af13-a7df13651950-kube-api-access-xtr69\") pod \"maas-api-revoke-sub-ce6f2b33-gq47k\" (UID: \"0e9392c8-52fd-48f2-af13-a7df13651950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" Oct 30 20:59:37.697604 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.697551 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9392c8-52fd-48f2-af13-a7df13651950-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce6f2b33-gq47k\" (UID: \"0e9392c8-52fd-48f2-af13-a7df13651950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" Oct 30 20:59:37.698080 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.698052 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91abf06-f896-4a66-9653-1e7533769692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c641beaf-nqd9j\" (UID: \"f91abf06-f896-4a66-9653-1e7533769692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" Oct 30 20:59:37.699883 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.699855 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91abf06-f896-4a66-9653-1e7533769692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c641beaf-nqd9j\" (UID: \"f91abf06-f896-4a66-9653-1e7533769692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" Oct 30 20:59:37.758566 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.758538 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7djf\" (UniqueName: \"kubernetes.io/projected/f91abf06-f896-4a66-9653-1e7533769692-kube-api-access-d7djf\") pod \"maas-api-revoke-sub-c641beaf-nqd9j\" (UID: \"f91abf06-f896-4a66-9653-1e7533769692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" Oct 30 20:59:37.799030 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.798958 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtr69\" (UniqueName: \"kubernetes.io/projected/0e9392c8-52fd-48f2-af13-a7df13651950-kube-api-access-xtr69\") pod \"maas-api-revoke-sub-ce6f2b33-gq47k\" (UID: \"0e9392c8-52fd-48f2-af13-a7df13651950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" Oct 30 20:59:37.799030 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.799004 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9392c8-52fd-48f2-af13-a7df13651950-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce6f2b33-gq47k\" (UID: \"0e9392c8-52fd-48f2-af13-a7df13651950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" Oct 30 20:59:37.799227 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.799063 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9392c8-52fd-48f2-af13-a7df13651950-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce6f2b33-gq47k\" (UID: \"0e9392c8-52fd-48f2-af13-a7df13651950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" Oct 30 20:59:37.799806 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.799727 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9392c8-52fd-48f2-af13-a7df13651950-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce6f2b33-gq47k\" (UID: \"0e9392c8-52fd-48f2-af13-a7df13651950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" Oct 30 20:59:37.801323 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.801302 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9392c8-52fd-48f2-af13-a7df13651950-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce6f2b33-gq47k\" (UID: \"0e9392c8-52fd-48f2-af13-a7df13651950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" Oct 30 20:59:37.856690 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.856661 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtr69\" (UniqueName: \"kubernetes.io/projected/0e9392c8-52fd-48f2-af13-a7df13651950-kube-api-access-xtr69\") pod \"maas-api-revoke-sub-ce6f2b33-gq47k\" (UID: \"0e9392c8-52fd-48f2-af13-a7df13651950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" Oct 30 20:59:37.864249 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.864220 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" Oct 30 20:59:37.922795 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:37.922762 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" Oct 30 20:59:38.238426 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:38.238398 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j"] Oct 30 20:59:38.239086 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:59:38.239063 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91abf06_f896_4a66_9653_1e7533769692.slice/crio-314693786ca48597d14f168bbfd615fec56e5e43e2fd06043145687fe5f1f0a6 WatchSource:0}: Error finding container 314693786ca48597d14f168bbfd615fec56e5e43e2fd06043145687fe5f1f0a6: Status 404 returned error can't find the container with id 314693786ca48597d14f168bbfd615fec56e5e43e2fd06043145687fe5f1f0a6 Oct 30 20:59:38.337939 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:38.337916 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k"] Oct 30 20:59:38.338873 ip-10-0-139-195 kubenswrapper[5755]: W1030 20:59:38.338847 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9392c8_52fd_48f2_af13_a7df13651950.slice/crio-221f5abcff2d3efe4cb39995e1c2541e7090818bb017f0ab86223cf88e49d29a WatchSource:0}: Error finding container 221f5abcff2d3efe4cb39995e1c2541e7090818bb017f0ab86223cf88e49d29a: Status 404 returned error can't find the container with id 221f5abcff2d3efe4cb39995e1c2541e7090818bb017f0ab86223cf88e49d29a Oct 30 20:59:38.598980 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:38.598948 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" event={"ID":"0e9392c8-52fd-48f2-af13-a7df13651950","Type":"ContainerStarted","Data":"68f7f8d803a89f51a447ccedb9dba3a6322b1cf2c41c9b7958dcc5aced4efbc3"} Oct 30 20:59:38.599362 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:38.599013 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" event={"ID":"0e9392c8-52fd-48f2-af13-a7df13651950","Type":"ContainerStarted","Data":"221f5abcff2d3efe4cb39995e1c2541e7090818bb017f0ab86223cf88e49d29a"} Oct 30 20:59:38.600651 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:38.600619 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" event={"ID":"f91abf06-f896-4a66-9653-1e7533769692","Type":"ContainerStarted","Data":"e3fd9fffbec67abc63f7224924322bf787a2eef20e5602ddd25678cd13a3ce3e"} Oct 30 20:59:38.600757 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:38.600672 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" event={"ID":"f91abf06-f896-4a66-9653-1e7533769692","Type":"ContainerStarted","Data":"314693786ca48597d14f168bbfd615fec56e5e43e2fd06043145687fe5f1f0a6"} Oct 30 20:59:38.719416 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:38.719359 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" podStartSLOduration=1.7193453010000002 podStartE2EDuration="1.719345301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:38.717492918 +0000 UTC m=+1941.644049848" watchObservedRunningTime="2026-10-30 20:59:38.719345301 +0000 UTC m=+1941.645902280" Oct 30 20:59:38.831627 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:38.831522 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" podStartSLOduration=1.831504155 podStartE2EDuration="1.831504155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:38.830337322 +0000 UTC m=+1941.756894237" watchObservedRunningTime="2026-10-30 20:59:38.831504155 +0000 UTC m=+1941.758061072" Oct 30 20:59:39.608732 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:39.608693 5755 generic.go:356] "Generic (PLEG): container finished" podID="0e9392c8-52fd-48f2-af13-a7df13651950" containerID="68f7f8d803a89f51a447ccedb9dba3a6322b1cf2c41c9b7958dcc5aced4efbc3" exitCode=0 Oct 30 20:59:39.609177 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:39.608773 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" event={"ID":"0e9392c8-52fd-48f2-af13-a7df13651950","Type":"ContainerDied","Data":"68f7f8d803a89f51a447ccedb9dba3a6322b1cf2c41c9b7958dcc5aced4efbc3"} Oct 30 20:59:39.610499 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:39.610475 5755 generic.go:356] "Generic (PLEG): container finished" podID="f91abf06-f896-4a66-9653-1e7533769692" containerID="e3fd9fffbec67abc63f7224924322bf787a2eef20e5602ddd25678cd13a3ce3e" exitCode=0 Oct 30 20:59:39.610616 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:39.610554 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" event={"ID":"f91abf06-f896-4a66-9653-1e7533769692","Type":"ContainerDied","Data":"e3fd9fffbec67abc63f7224924322bf787a2eef20e5602ddd25678cd13a3ce3e"} Oct 30 20:59:39.824953 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:39.824921 5755 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:59:39.825126 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:39.824980 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" podUID="dec4885f-2622-40b5-bc12-e15f5ec1e51c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:59:40.376710 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.376687 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:59:40.431365 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.431300 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec4885f-2622-40b5-bc12-e15f5ec1e51c-kube-api-access-5qqzv\" (UniqueName: \"kubernetes.io/projected/dec4885f-2622-40b5-bc12-e15f5ec1e51c-kube-api-access-5qqzv\") pod \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " Oct 30 20:59:40.431365 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.431355 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istiod-ca-cert\") pod \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " Oct 30 20:59:40.431592 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.431419 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-workload-socket\") pod \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " Oct 30 20:59:40.431592 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.431466 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-credential-socket\") pod \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " Oct 30 20:59:40.431592 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.431519 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-token\" (UniqueName: \"kubernetes.io/projected/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-token\") pod \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " Oct 30 20:59:40.431592 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.431559 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-workload-certs\") pod \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " Oct 30 20:59:40.431802 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.431681 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-podinfo\") pod \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " Oct 30 20:59:40.431802 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.431723 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-data\") pod \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " Oct 30 20:59:40.431802 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.431784 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-envoy\") pod \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\" (UID: \"dec4885f-2622-40b5-bc12-e15f5ec1e51c\") " Oct 30 20:59:40.432033 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.431780 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-workload-socket" pod "dec4885f-2622-40b5-bc12-e15f5ec1e51c" (UID: "dec4885f-2622-40b5-bc12-e15f5ec1e51c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:40.432033 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.431793 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-credential-socket" pod "dec4885f-2622-40b5-bc12-e15f5ec1e51c" (UID: "dec4885f-2622-40b5-bc12-e15f5ec1e51c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:40.432033 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.431814 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istiod-ca-cert" pod "dec4885f-2622-40b5-bc12-e15f5ec1e51c" (UID: "dec4885f-2622-40b5-bc12-e15f5ec1e51c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:40.432033 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.431928 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-workload-certs" pod "dec4885f-2622-40b5-bc12-e15f5ec1e51c" (UID: "dec4885f-2622-40b5-bc12-e15f5ec1e51c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:40.432033 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.431982 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-data" pod "dec4885f-2622-40b5-bc12-e15f5ec1e51c" (UID: "dec4885f-2622-40b5-bc12-e15f5ec1e51c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:40.432588 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.432197 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.432588 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.432216 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.432588 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.432230 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.432588 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.432240 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.432588 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.432251 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.434091 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.434042 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-podinfo" pod "dec4885f-2622-40b5-bc12-e15f5ec1e51c" (UID: "dec4885f-2622-40b5-bc12-e15f5ec1e51c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:59:40.434198 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.434136 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-token" pod "dec4885f-2622-40b5-bc12-e15f5ec1e51c" (UID: "dec4885f-2622-40b5-bc12-e15f5ec1e51c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:40.434453 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.434428 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-envoy" pod "dec4885f-2622-40b5-bc12-e15f5ec1e51c" (UID: "dec4885f-2622-40b5-bc12-e15f5ec1e51c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:40.434453 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.434440 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec4885f-2622-40b5-bc12-e15f5ec1e51c-kube-api-access-5qqzv" pod "dec4885f-2622-40b5-bc12-e15f5ec1e51c" (UID: "dec4885f-2622-40b5-bc12-e15f5ec1e51c"). InnerVolumeSpecName "kube-api-access-5qqzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:40.533904 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.533867 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.533904 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.533905 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.534097 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.533920 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qqzv\" (UniqueName: \"kubernetes.io/projected/dec4885f-2622-40b5-bc12-e15f5ec1e51c-kube-api-access-5qqzv\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.534097 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.533938 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec4885f-2622-40b5-bc12-e15f5ec1e51c-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.618325 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.618290 5755 generic.go:356] "Generic (PLEG): container finished" podID="dec4885f-2622-40b5-bc12-e15f5ec1e51c" containerID="3a257893ff509f695560ca9a3dbd1820eb46808f2f3ae64ab1dadb8bf6e9fc40" exitCode=0 Oct 30 20:59:40.618835 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.618334 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" Oct 30 20:59:40.618835 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.618381 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" event={"ID":"dec4885f-2622-40b5-bc12-e15f5ec1e51c","Type":"ContainerDied","Data":"3a257893ff509f695560ca9a3dbd1820eb46808f2f3ae64ab1dadb8bf6e9fc40"} Oct 30 20:59:40.618835 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.618479 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6" event={"ID":"dec4885f-2622-40b5-bc12-e15f5ec1e51c","Type":"ContainerDied","Data":"da8b6837d1f7f49b59694ef2148f8976ca4d26e8083df32e469c5e436c509073"} Oct 30 20:59:40.618835 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.618515 5755 scope.go:122] "RemoveContainer" containerID="3a257893ff509f695560ca9a3dbd1820eb46808f2f3ae64ab1dadb8bf6e9fc40" Oct 30 20:59:40.630407 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.630358 5755 scope.go:122] "RemoveContainer" containerID="3a257893ff509f695560ca9a3dbd1820eb46808f2f3ae64ab1dadb8bf6e9fc40" Oct 30 20:59:40.630708 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:59:40.630684 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a257893ff509f695560ca9a3dbd1820eb46808f2f3ae64ab1dadb8bf6e9fc40\": container with ID starting with 3a257893ff509f695560ca9a3dbd1820eb46808f2f3ae64ab1dadb8bf6e9fc40 not found: ID does not exist" containerID="3a257893ff509f695560ca9a3dbd1820eb46808f2f3ae64ab1dadb8bf6e9fc40" Oct 30 20:59:40.630778 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.630717 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a257893ff509f695560ca9a3dbd1820eb46808f2f3ae64ab1dadb8bf6e9fc40"} err="failed to get container status \"3a257893ff509f695560ca9a3dbd1820eb46808f2f3ae64ab1dadb8bf6e9fc40\": rpc error: code = NotFound desc = could not find container \"3a257893ff509f695560ca9a3dbd1820eb46808f2f3ae64ab1dadb8bf6e9fc40\": container with ID starting with 3a257893ff509f695560ca9a3dbd1820eb46808f2f3ae64ab1dadb8bf6e9fc40 not found: ID does not exist" Oct 30 20:59:40.756611 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.756575 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" Oct 30 20:59:40.764174 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.764152 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" Oct 30 20:59:40.786063 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.786040 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6"] Oct 30 20:59:40.791021 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.790998 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-ds5d6"] Oct 30 20:59:40.837749 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.837718 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91abf06-f896-4a66-9653-1e7533769692-kube-api-access-d7djf\" (UniqueName: \"kubernetes.io/projected/f91abf06-f896-4a66-9653-1e7533769692-kube-api-access-d7djf\") pod \"f91abf06-f896-4a66-9653-1e7533769692\" (UID: \"f91abf06-f896-4a66-9653-1e7533769692\") " Oct 30 20:59:40.837934 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.837763 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f91abf06-f896-4a66-9653-1e7533769692-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91abf06-f896-4a66-9653-1e7533769692-maas-api-openshift-ca\") pod \"f91abf06-f896-4a66-9653-1e7533769692\" (UID: \"f91abf06-f896-4a66-9653-1e7533769692\") " Oct 30 20:59:40.837934 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.837797 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e9392c8-52fd-48f2-af13-a7df13651950-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9392c8-52fd-48f2-af13-a7df13651950-maas-api-xks-ca\") pod \"0e9392c8-52fd-48f2-af13-a7df13651950\" (UID: \"0e9392c8-52fd-48f2-af13-a7df13651950\") " Oct 30 20:59:40.837934 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.837836 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9392c8-52fd-48f2-af13-a7df13651950-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9392c8-52fd-48f2-af13-a7df13651950-maas-api-openshift-ca\") pod \"0e9392c8-52fd-48f2-af13-a7df13651950\" (UID: \"0e9392c8-52fd-48f2-af13-a7df13651950\") " Oct 30 20:59:40.838089 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.838067 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9392c8-52fd-48f2-af13-a7df13651950-kube-api-access-xtr69\" (UniqueName: \"kubernetes.io/projected/0e9392c8-52fd-48f2-af13-a7df13651950-kube-api-access-xtr69\") pod \"0e9392c8-52fd-48f2-af13-a7df13651950\" (UID: \"0e9392c8-52fd-48f2-af13-a7df13651950\") " Oct 30 20:59:40.838145 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.838103 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f91abf06-f896-4a66-9653-1e7533769692-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91abf06-f896-4a66-9653-1e7533769692-maas-api-xks-ca\") pod \"f91abf06-f896-4a66-9653-1e7533769692\" (UID: \"f91abf06-f896-4a66-9653-1e7533769692\") " Oct 30 20:59:40.838196 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.838172 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91abf06-f896-4a66-9653-1e7533769692-maas-api-openshift-ca" pod "f91abf06-f896-4a66-9653-1e7533769692" (UID: "f91abf06-f896-4a66-9653-1e7533769692"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:40.838273 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.838231 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9392c8-52fd-48f2-af13-a7df13651950-maas-api-openshift-ca" pod "0e9392c8-52fd-48f2-af13-a7df13651950" (UID: "0e9392c8-52fd-48f2-af13-a7df13651950"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:40.838815 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.838790 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91abf06-f896-4a66-9653-1e7533769692-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.838925 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.838821 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9392c8-52fd-48f2-af13-a7df13651950-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.840250 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.840220 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9392c8-52fd-48f2-af13-a7df13651950-maas-api-xks-ca" pod "0e9392c8-52fd-48f2-af13-a7df13651950" (UID: "0e9392c8-52fd-48f2-af13-a7df13651950"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:40.840348 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.840287 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f91abf06-f896-4a66-9653-1e7533769692-maas-api-xks-ca" pod "f91abf06-f896-4a66-9653-1e7533769692" (UID: "f91abf06-f896-4a66-9653-1e7533769692"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:40.840348 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.840311 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91abf06-f896-4a66-9653-1e7533769692-kube-api-access-d7djf" pod "f91abf06-f896-4a66-9653-1e7533769692" (UID: "f91abf06-f896-4a66-9653-1e7533769692"). InnerVolumeSpecName "kube-api-access-d7djf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:40.840348 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.840339 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9392c8-52fd-48f2-af13-a7df13651950-kube-api-access-xtr69" pod "0e9392c8-52fd-48f2-af13-a7df13651950" (UID: "0e9392c8-52fd-48f2-af13-a7df13651950"). InnerVolumeSpecName "kube-api-access-xtr69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:40.940304 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.940272 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7djf\" (UniqueName: \"kubernetes.io/projected/f91abf06-f896-4a66-9653-1e7533769692-kube-api-access-d7djf\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.940304 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.940307 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9392c8-52fd-48f2-af13-a7df13651950-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.940541 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.940321 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtr69\" (UniqueName: \"kubernetes.io/projected/0e9392c8-52fd-48f2-af13-a7df13651950-kube-api-access-xtr69\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.940541 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:40.940336 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91abf06-f896-4a66-9653-1e7533769692-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:41.075078 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:41.075038 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt"] Oct 30 20:59:41.075329 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:41.075307 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" podUID="b830f9c4-8e19-4ee1-9bd3-e41b667a9122" containerName="istio-proxy" containerID="cri-o://60136f3eb20c6b89dc276d41b6a4e1310612431c8428e20d7fb0767373993c0f" gracePeriod=30 Oct 30 20:59:41.217808 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:41.217771 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec4885f-2622-40b5-bc12-e15f5ec1e51c" path="/var/lib/kubelet/pods/dec4885f-2622-40b5-bc12-e15f5ec1e51c/volumes" Oct 30 20:59:41.255130 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:59:41.255097 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 20:59:41.627316 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:41.627290 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" Oct 30 20:59:41.627804 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:41.627290 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k" event={"ID":"0e9392c8-52fd-48f2-af13-a7df13651950","Type":"ContainerDied","Data":"221f5abcff2d3efe4cb39995e1c2541e7090818bb017f0ab86223cf88e49d29a"} Oct 30 20:59:41.627804 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:41.627447 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="221f5abcff2d3efe4cb39995e1c2541e7090818bb017f0ab86223cf88e49d29a" Oct 30 20:59:41.628976 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:41.628955 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" Oct 30 20:59:41.629084 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:41.628957 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j" event={"ID":"f91abf06-f896-4a66-9653-1e7533769692","Type":"ContainerDied","Data":"314693786ca48597d14f168bbfd615fec56e5e43e2fd06043145687fe5f1f0a6"} Oct 30 20:59:41.629084 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:41.629078 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="314693786ca48597d14f168bbfd615fec56e5e43e2fd06043145687fe5f1f0a6" Oct 30 20:59:41.681312 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:41.681288 5755 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:59:41.681476 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:41.681327 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" podUID="b830f9c4-8e19-4ee1-9bd3-e41b667a9122" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:59:44.911245 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:44.911211 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv"] Oct 30 20:59:44.915032 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:44.915007 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-9e0fb1-dcqpv"] Oct 30 20:59:45.212097 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:45.212019 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="214c6ae5-0dfe-48b0-b871-403167878a51" path="/var/lib/kubelet/pods/214c6ae5-0dfe-48b0-b871-403167878a51/volumes" Oct 30 20:59:46.340867 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.340838 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:59:46.414558 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.414490 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istiod-ca-cert\") pod \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " Oct 30 20:59:46.414716 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.414603 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-data\") pod \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " Oct 30 20:59:46.414716 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.414638 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-token\" (UniqueName: \"kubernetes.io/projected/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-token\") pod \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " Oct 30 20:59:46.414716 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.414699 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-podinfo\") pod \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " Oct 30 20:59:46.414886 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.414747 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-envoy\") pod \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " Oct 30 20:59:46.414886 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.414789 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-workload-socket\") pod \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " Oct 30 20:59:46.414886 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.414845 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-workload-certs\") pod \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " Oct 30 20:59:46.415041 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.414875 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-data" pod "b830f9c4-8e19-4ee1-9bd3-e41b667a9122" (UID: "b830f9c4-8e19-4ee1-9bd3-e41b667a9122"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:46.415041 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.414890 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istiod-ca-cert" pod "b830f9c4-8e19-4ee1-9bd3-e41b667a9122" (UID: "b830f9c4-8e19-4ee1-9bd3-e41b667a9122"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:46.415041 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.414905 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-kube-api-access-56r5p\" (UniqueName: \"kubernetes.io/projected/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-kube-api-access-56r5p\") pod \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " Oct 30 20:59:46.415041 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.414966 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-credential-socket\") pod \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\" (UID: \"b830f9c4-8e19-4ee1-9bd3-e41b667a9122\") " Oct 30 20:59:46.415231 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.415114 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-workload-socket" pod "b830f9c4-8e19-4ee1-9bd3-e41b667a9122" (UID: "b830f9c4-8e19-4ee1-9bd3-e41b667a9122"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:46.415231 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.415167 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-workload-certs" pod "b830f9c4-8e19-4ee1-9bd3-e41b667a9122" (UID: "b830f9c4-8e19-4ee1-9bd3-e41b667a9122"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:46.415355 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.415335 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-credential-socket" pod "b830f9c4-8e19-4ee1-9bd3-e41b667a9122" (UID: "b830f9c4-8e19-4ee1-9bd3-e41b667a9122"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:46.415521 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.415402 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:46.415595 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.415532 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:46.415595 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.415544 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:46.415595 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.415554 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:46.417616 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.417492 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-envoy" pod "b830f9c4-8e19-4ee1-9bd3-e41b667a9122" (UID: "b830f9c4-8e19-4ee1-9bd3-e41b667a9122"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:46.417616 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.417564 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-podinfo" pod "b830f9c4-8e19-4ee1-9bd3-e41b667a9122" (UID: "b830f9c4-8e19-4ee1-9bd3-e41b667a9122"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:59:46.417616 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.417586 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-token" pod "b830f9c4-8e19-4ee1-9bd3-e41b667a9122" (UID: "b830f9c4-8e19-4ee1-9bd3-e41b667a9122"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:46.417616 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.417597 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-kube-api-access-56r5p" pod "b830f9c4-8e19-4ee1-9bd3-e41b667a9122" (UID: "b830f9c4-8e19-4ee1-9bd3-e41b667a9122"). InnerVolumeSpecName "kube-api-access-56r5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:46.516614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.516578 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:46.516614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.516613 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:46.516864 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.516627 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:46.516864 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.516642 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56r5p\" (UniqueName: \"kubernetes.io/projected/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-kube-api-access-56r5p\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:46.516864 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.516658 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b830f9c4-8e19-4ee1-9bd3-e41b667a9122-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:46.670675 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.670576 5755 generic.go:356] "Generic (PLEG): container finished" podID="b830f9c4-8e19-4ee1-9bd3-e41b667a9122" containerID="60136f3eb20c6b89dc276d41b6a4e1310612431c8428e20d7fb0767373993c0f" exitCode=0 Oct 30 20:59:46.670675 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.670643 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" Oct 30 20:59:46.670675 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.670661 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" event={"ID":"b830f9c4-8e19-4ee1-9bd3-e41b667a9122","Type":"ContainerDied","Data":"60136f3eb20c6b89dc276d41b6a4e1310612431c8428e20d7fb0767373993c0f"} Oct 30 20:59:46.670982 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.670754 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt" event={"ID":"b830f9c4-8e19-4ee1-9bd3-e41b667a9122","Type":"ContainerDied","Data":"9b49caf69e00b5462fde5fad6a8c0260e12e6a418b750d3debb14d5a4e0011ce"} Oct 30 20:59:46.670982 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.670782 5755 scope.go:122] "RemoveContainer" containerID="60136f3eb20c6b89dc276d41b6a4e1310612431c8428e20d7fb0767373993c0f" Oct 30 20:59:46.683110 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.683089 5755 scope.go:122] "RemoveContainer" containerID="60136f3eb20c6b89dc276d41b6a4e1310612431c8428e20d7fb0767373993c0f" Oct 30 20:59:46.683837 ip-10-0-139-195 kubenswrapper[5755]: E1030 20:59:46.683814 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60136f3eb20c6b89dc276d41b6a4e1310612431c8428e20d7fb0767373993c0f\": container with ID starting with 60136f3eb20c6b89dc276d41b6a4e1310612431c8428e20d7fb0767373993c0f not found: ID does not exist" containerID="60136f3eb20c6b89dc276d41b6a4e1310612431c8428e20d7fb0767373993c0f" Oct 30 20:59:46.683927 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.683845 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60136f3eb20c6b89dc276d41b6a4e1310612431c8428e20d7fb0767373993c0f"} err="failed to get container status \"60136f3eb20c6b89dc276d41b6a4e1310612431c8428e20d7fb0767373993c0f\": rpc error: code = NotFound desc = could not find container \"60136f3eb20c6b89dc276d41b6a4e1310612431c8428e20d7fb0767373993c0f\": container with ID starting with 60136f3eb20c6b89dc276d41b6a4e1310612431c8428e20d7fb0767373993c0f not found: ID does not exist" Oct 30 20:59:46.838831 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.838789 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt"] Oct 30 20:59:46.843879 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:46.843848 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-d6qwt"] Oct 30 20:59:47.212017 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:47.211986 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b830f9c4-8e19-4ee1-9bd3-e41b667a9122" path="/var/lib/kubelet/pods/b830f9c4-8e19-4ee1-9bd3-e41b667a9122/volumes" Oct 30 20:59:50.486303 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:50.486273 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf"] Oct 30 20:59:50.486735 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:50.486506 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" podUID="81547e38-6ed0-4a54-bdec-8972a18f2f1a" containerName="istio-proxy" containerID="cri-o://d0d218d7870b7fa938e4a1cb95568c4452b400f577f8e53ce85b3268543a7c09" gracePeriod=30 Oct 30 20:59:51.186266 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:51.186233 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp"] Oct 30 20:59:51.235831 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:51.235801 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d9b3f6-794mp"] Oct 30 20:59:53.217571 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:53.217538 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5601e3e-ded9-4692-8a4f-0ff0335e8c98" path="/var/lib/kubelet/pods/a5601e3e-ded9-4692-8a4f-0ff0335e8c98/volumes" Oct 30 20:59:55.346070 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.346035 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7"] Oct 30 20:59:55.346512 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.346238 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" podUID="4e15faa4-f00e-4316-9b78-fe0b40a09ac2" containerName="istio-proxy" containerID="cri-o://f9533248ebd674201f131e65b0660472914b651df7c2d52856233e47f6257793" gracePeriod=30 Oct 30 20:59:55.397051 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.397018 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252"] Oct 30 20:59:55.397295 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.397261 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" podUID="2965df53-329d-4971-893e-3c60b72762c5" containerName="istio-proxy" containerID="cri-o://7c06707a7b2b6d9a3683dcecab74d71866c7ce33e48e986ca22021dfd3218d33" gracePeriod=30 Oct 30 20:59:55.732913 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.732885 5755 generic.go:356] "Generic (PLEG): container finished" podID="81547e38-6ed0-4a54-bdec-8972a18f2f1a" containerID="d0d218d7870b7fa938e4a1cb95568c4452b400f577f8e53ce85b3268543a7c09" exitCode=0 Oct 30 20:59:55.733039 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.732933 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" event={"ID":"81547e38-6ed0-4a54-bdec-8972a18f2f1a","Type":"ContainerDied","Data":"d0d218d7870b7fa938e4a1cb95568c4452b400f577f8e53ce85b3268543a7c09"} Oct 30 20:59:55.733039 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.732973 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" event={"ID":"81547e38-6ed0-4a54-bdec-8972a18f2f1a","Type":"ContainerDied","Data":"a4ebbac5a7d67ac63c18a6a686d259e29204f3a0f1d9fe802d1f575645d8f67e"} Oct 30 20:59:55.733039 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.732990 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4ebbac5a7d67ac63c18a6a686d259e29204f3a0f1d9fe802d1f575645d8f67e" Oct 30 20:59:55.740049 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.740027 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:59:55.752345 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.752318 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-credential-socket\") pod \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " Oct 30 20:59:55.752557 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.752416 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istiod-ca-cert\") pod \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " Oct 30 20:59:55.752557 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.752468 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81547e38-6ed0-4a54-bdec-8972a18f2f1a-kube-api-access-h4rjf\" (UniqueName: \"kubernetes.io/projected/81547e38-6ed0-4a54-bdec-8972a18f2f1a-kube-api-access-h4rjf\") pod \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " Oct 30 20:59:55.752557 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.752529 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-data\") pod \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " Oct 30 20:59:55.752802 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.752569 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-podinfo\") pod \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " Oct 30 20:59:55.752802 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.752622 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-envoy\") pod \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " Oct 30 20:59:55.752802 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.752641 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-credential-socket" pod "81547e38-6ed0-4a54-bdec-8972a18f2f1a" (UID: "81547e38-6ed0-4a54-bdec-8972a18f2f1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:55.752802 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.752663 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-workload-certs\") pod \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " Oct 30 20:59:55.752802 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.752692 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-workload-socket\") pod \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " Oct 30 20:59:55.752802 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.752742 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-token\" (UniqueName: \"kubernetes.io/projected/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-token\") pod \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\" (UID: \"81547e38-6ed0-4a54-bdec-8972a18f2f1a\") " Oct 30 20:59:55.753105 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.753063 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istiod-ca-cert" pod "81547e38-6ed0-4a54-bdec-8972a18f2f1a" (UID: "81547e38-6ed0-4a54-bdec-8972a18f2f1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:55.753207 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.753185 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:55.753327 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.753211 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:55.753559 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.753537 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-workload-socket" pod "81547e38-6ed0-4a54-bdec-8972a18f2f1a" (UID: "81547e38-6ed0-4a54-bdec-8972a18f2f1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:55.753661 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.753562 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-workload-certs" pod "81547e38-6ed0-4a54-bdec-8972a18f2f1a" (UID: "81547e38-6ed0-4a54-bdec-8972a18f2f1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:55.753661 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.753608 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-data" pod "81547e38-6ed0-4a54-bdec-8972a18f2f1a" (UID: "81547e38-6ed0-4a54-bdec-8972a18f2f1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:55.755191 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.755166 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-token" pod "81547e38-6ed0-4a54-bdec-8972a18f2f1a" (UID: "81547e38-6ed0-4a54-bdec-8972a18f2f1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:55.755614 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.755588 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-podinfo" pod "81547e38-6ed0-4a54-bdec-8972a18f2f1a" (UID: "81547e38-6ed0-4a54-bdec-8972a18f2f1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:59:55.755710 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.755655 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-envoy" pod "81547e38-6ed0-4a54-bdec-8972a18f2f1a" (UID: "81547e38-6ed0-4a54-bdec-8972a18f2f1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:55.755710 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.755665 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81547e38-6ed0-4a54-bdec-8972a18f2f1a-kube-api-access-h4rjf" pod "81547e38-6ed0-4a54-bdec-8972a18f2f1a" (UID: "81547e38-6ed0-4a54-bdec-8972a18f2f1a"). InnerVolumeSpecName "kube-api-access-h4rjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:55.854255 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.854185 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4rjf\" (UniqueName: \"kubernetes.io/projected/81547e38-6ed0-4a54-bdec-8972a18f2f1a-kube-api-access-h4rjf\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:55.854255 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.854213 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:55.854255 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.854224 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:55.854255 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.854232 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:55.854255 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.854241 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:55.854255 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.854249 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81547e38-6ed0-4a54-bdec-8972a18f2f1a-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:55.854255 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:55.854257 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81547e38-6ed0-4a54-bdec-8972a18f2f1a-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:56.739774 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:56.739745 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf" Oct 30 20:59:56.908985 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:56.908945 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf"] Oct 30 20:59:56.958503 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:56.958470 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-brmdf"] Oct 30 20:59:57.211420 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:57.211295 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81547e38-6ed0-4a54-bdec-8972a18f2f1a" path="/var/lib/kubelet/pods/81547e38-6ed0-4a54-bdec-8972a18f2f1a/volumes" Oct 30 20:59:59.528636 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:59.528606 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9"] Oct 30 20:59:59.578271 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:59.578240 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445"] Oct 30 20:59:59.580665 ip-10-0-139-195 kubenswrapper[5755]: I1030 20:59:59.580615 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f"] Oct 30 21:00:00.166500 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.166467 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv"] Oct 30 21:00:00.166938 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.166921 5755 cpu_manager.go:414] "containerMap: removing container" podUID="b830f9c4-8e19-4ee1-9bd3-e41b667a9122" containerName="istio-proxy" Oct 30 21:00:00.166938 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.166938 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b830f9c4-8e19-4ee1-9bd3-e41b667a9122" containerName="istio-proxy" Oct 30 21:00:00.167069 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.166958 5755 cpu_manager.go:414] "containerMap: removing container" podUID="f91abf06-f896-4a66-9653-1e7533769692" containerName="revoke-keys" Oct 30 21:00:00.167069 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.166964 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91abf06-f896-4a66-9653-1e7533769692" containerName="revoke-keys" Oct 30 21:00:00.167069 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.166983 5755 cpu_manager.go:414] "containerMap: removing container" podUID="0e9392c8-52fd-48f2-af13-a7df13651950" containerName="revoke-keys" Oct 30 21:00:00.167069 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.166991 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9392c8-52fd-48f2-af13-a7df13651950" containerName="revoke-keys" Oct 30 21:00:00.167069 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.167011 5755 cpu_manager.go:414] "containerMap: removing container" podUID="81547e38-6ed0-4a54-bdec-8972a18f2f1a" containerName="istio-proxy" Oct 30 21:00:00.167069 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.167016 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81547e38-6ed0-4a54-bdec-8972a18f2f1a" containerName="istio-proxy" Oct 30 21:00:00.167069 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.167029 5755 cpu_manager.go:414] "containerMap: removing container" podUID="dec4885f-2622-40b5-bc12-e15f5ec1e51c" containerName="istio-proxy" Oct 30 21:00:00.167069 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.167033 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec4885f-2622-40b5-bc12-e15f5ec1e51c" containerName="istio-proxy" Oct 30 21:00:00.167299 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.167098 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="81547e38-6ed0-4a54-bdec-8972a18f2f1a" containerName="istio-proxy" Oct 30 21:00:00.167299 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.167108 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec4885f-2622-40b5-bc12-e15f5ec1e51c" containerName="istio-proxy" Oct 30 21:00:00.167299 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.167114 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="b830f9c4-8e19-4ee1-9bd3-e41b667a9122" containerName="istio-proxy" Oct 30 21:00:00.167299 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.167120 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91abf06-f896-4a66-9653-1e7533769692" containerName="revoke-keys" Oct 30 21:00:00.167299 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.167126 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9392c8-52fd-48f2-af13-a7df13651950" containerName="revoke-keys" Oct 30 21:00:00.171556 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.171537 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj"] Oct 30 21:00:00.171669 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.171648 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" Oct 30 21:00:00.175230 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.175214 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" Oct 30 21:00:00.199325 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.199304 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lfnk\" (UniqueName: \"kubernetes.io/projected/af12be6a-d0d3-47f5-bce7-65439f599fe6-kube-api-access-7lfnk\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv\" (UID: \"af12be6a-d0d3-47f5-bce7-65439f599fe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" Oct 30 21:00:00.199446 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.199362 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2cpt\" (UniqueName: \"kubernetes.io/projected/41470888-65ed-44de-9bcc-210cf168dece-kube-api-access-r2cpt\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj\" (UID: \"41470888-65ed-44de-9bcc-210cf168dece\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" Oct 30 21:00:00.199492 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.199466 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af12be6a-d0d3-47f5-bce7-65439f599fe6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv\" (UID: \"af12be6a-d0d3-47f5-bce7-65439f599fe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" Oct 30 21:00:00.199565 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.199553 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41470888-65ed-44de-9bcc-210cf168dece-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj\" (UID: \"41470888-65ed-44de-9bcc-210cf168dece\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" Oct 30 21:00:00.229933 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.229910 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 21:00:00.230266 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.230250 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv"] Oct 30 21:00:00.234556 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.234535 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj"] Oct 30 21:00:00.300254 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.300226 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2cpt\" (UniqueName: \"kubernetes.io/projected/41470888-65ed-44de-9bcc-210cf168dece-kube-api-access-r2cpt\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj\" (UID: \"41470888-65ed-44de-9bcc-210cf168dece\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" Oct 30 21:00:00.300423 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.300261 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af12be6a-d0d3-47f5-bce7-65439f599fe6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv\" (UID: \"af12be6a-d0d3-47f5-bce7-65439f599fe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" Oct 30 21:00:00.300538 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.300514 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41470888-65ed-44de-9bcc-210cf168dece-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj\" (UID: \"41470888-65ed-44de-9bcc-210cf168dece\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" Oct 30 21:00:00.300617 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.300591 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lfnk\" (UniqueName: \"kubernetes.io/projected/af12be6a-d0d3-47f5-bce7-65439f599fe6-kube-api-access-7lfnk\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv\" (UID: \"af12be6a-d0d3-47f5-bce7-65439f599fe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" Oct 30 21:00:00.301028 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.301012 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af12be6a-d0d3-47f5-bce7-65439f599fe6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv\" (UID: \"af12be6a-d0d3-47f5-bce7-65439f599fe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" Oct 30 21:00:00.301130 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.301111 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41470888-65ed-44de-9bcc-210cf168dece-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj\" (UID: \"41470888-65ed-44de-9bcc-210cf168dece\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" Oct 30 21:00:00.358843 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.358818 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lfnk\" (UniqueName: \"kubernetes.io/projected/af12be6a-d0d3-47f5-bce7-65439f599fe6-kube-api-access-7lfnk\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv\" (UID: \"af12be6a-d0d3-47f5-bce7-65439f599fe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" Oct 30 21:00:00.359032 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.359011 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2cpt\" (UniqueName: \"kubernetes.io/projected/41470888-65ed-44de-9bcc-210cf168dece-kube-api-access-r2cpt\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj\" (UID: \"41470888-65ed-44de-9bcc-210cf168dece\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" Oct 30 21:00:00.485054 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.485028 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" Oct 30 21:00:00.491068 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.491048 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" Oct 30 21:00:00.713959 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.713935 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 21:00:00.723427 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.723371 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 21:00:00.765365 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.765339 5755 generic.go:356] "Generic (PLEG): container finished" podID="4e15faa4-f00e-4316-9b78-fe0b40a09ac2" containerID="f9533248ebd674201f131e65b0660472914b651df7c2d52856233e47f6257793" exitCode=0 Oct 30 21:00:00.765530 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.765514 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" event={"ID":"4e15faa4-f00e-4316-9b78-fe0b40a09ac2","Type":"ContainerDied","Data":"f9533248ebd674201f131e65b0660472914b651df7c2d52856233e47f6257793"} Oct 30 21:00:00.765573 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.765558 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" event={"ID":"4e15faa4-f00e-4316-9b78-fe0b40a09ac2","Type":"ContainerDied","Data":"ad3673d277f4ebc0524c1fd0a2be3c49313496d85cab1ac97b8f8e928c64c4b4"} Oct 30 21:00:00.765608 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.765581 5755 scope.go:122] "RemoveContainer" containerID="f9533248ebd674201f131e65b0660472914b651df7c2d52856233e47f6257793" Oct 30 21:00:00.765697 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.765687 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7" Oct 30 21:00:00.767449 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.767425 5755 generic.go:356] "Generic (PLEG): container finished" podID="2965df53-329d-4971-893e-3c60b72762c5" containerID="7c06707a7b2b6d9a3683dcecab74d71866c7ce33e48e986ca22021dfd3218d33" exitCode=0 Oct 30 21:00:00.767572 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.767548 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" event={"ID":"2965df53-329d-4971-893e-3c60b72762c5","Type":"ContainerDied","Data":"7c06707a7b2b6d9a3683dcecab74d71866c7ce33e48e986ca22021dfd3218d33"} Oct 30 21:00:00.767643 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.767580 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" event={"ID":"2965df53-329d-4971-893e-3c60b72762c5","Type":"ContainerDied","Data":"2863087bc524a73f73b8fb02d71f934df594a398370c28df3ad94d0965070822"} Oct 30 21:00:00.767643 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.767638 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252" Oct 30 21:00:00.775679 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.775657 5755 scope.go:122] "RemoveContainer" containerID="f9533248ebd674201f131e65b0660472914b651df7c2d52856233e47f6257793" Oct 30 21:00:00.775929 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:00.775910 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9533248ebd674201f131e65b0660472914b651df7c2d52856233e47f6257793\": container with ID starting with f9533248ebd674201f131e65b0660472914b651df7c2d52856233e47f6257793 not found: ID does not exist" containerID="f9533248ebd674201f131e65b0660472914b651df7c2d52856233e47f6257793" Oct 30 21:00:00.776001 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.775955 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9533248ebd674201f131e65b0660472914b651df7c2d52856233e47f6257793"} err="failed to get container status \"f9533248ebd674201f131e65b0660472914b651df7c2d52856233e47f6257793\": rpc error: code = NotFound desc = could not find container \"f9533248ebd674201f131e65b0660472914b651df7c2d52856233e47f6257793\": container with ID starting with f9533248ebd674201f131e65b0660472914b651df7c2d52856233e47f6257793 not found: ID does not exist" Oct 30 21:00:00.776001 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.775980 5755 scope.go:122] "RemoveContainer" containerID="7c06707a7b2b6d9a3683dcecab74d71866c7ce33e48e986ca22021dfd3218d33" Oct 30 21:00:00.785205 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.785189 5755 scope.go:122] "RemoveContainer" containerID="7c06707a7b2b6d9a3683dcecab74d71866c7ce33e48e986ca22021dfd3218d33" Oct 30 21:00:00.785505 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:00.785487 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c06707a7b2b6d9a3683dcecab74d71866c7ce33e48e986ca22021dfd3218d33\": container with ID starting with 7c06707a7b2b6d9a3683dcecab74d71866c7ce33e48e986ca22021dfd3218d33 not found: ID does not exist" containerID="7c06707a7b2b6d9a3683dcecab74d71866c7ce33e48e986ca22021dfd3218d33" Oct 30 21:00:00.785564 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.785516 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c06707a7b2b6d9a3683dcecab74d71866c7ce33e48e986ca22021dfd3218d33"} err="failed to get container status \"7c06707a7b2b6d9a3683dcecab74d71866c7ce33e48e986ca22021dfd3218d33\": rpc error: code = NotFound desc = could not find container \"7c06707a7b2b6d9a3683dcecab74d71866c7ce33e48e986ca22021dfd3218d33\": container with ID starting with 7c06707a7b2b6d9a3683dcecab74d71866c7ce33e48e986ca22021dfd3218d33 not found: ID does not exist" Oct 30 21:00:00.804979 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.804954 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-workload-certs\") pod \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " Oct 30 21:00:00.805078 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.804997 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2965df53-329d-4971-893e-3c60b72762c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2965df53-329d-4971-893e-3c60b72762c5-istiod-ca-cert\") pod \"2965df53-329d-4971-893e-3c60b72762c5\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " Oct 30 21:00:00.805078 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805028 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-workload-certs\") pod \"2965df53-329d-4971-893e-3c60b72762c5\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " Oct 30 21:00:00.805191 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805083 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-credential-socket\") pod \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " Oct 30 21:00:00.805191 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805113 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-data\") pod \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " Oct 30 21:00:00.805191 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805144 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-istio-envoy\") pod \"2965df53-329d-4971-893e-3c60b72762c5\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " Oct 30 21:00:00.805191 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805174 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-workload-socket\") pod \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " Oct 30 21:00:00.805414 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805227 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-workload-certs" pod "4e15faa4-f00e-4316-9b78-fe0b40a09ac2" (UID: "4e15faa4-f00e-4316-9b78-fe0b40a09ac2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:00.805414 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805242 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-podinfo\") pod \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " Oct 30 21:00:00.805414 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805319 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2965df53-329d-4971-893e-3c60b72762c5-istio-token\" (UniqueName: \"kubernetes.io/projected/2965df53-329d-4971-893e-3c60b72762c5-istio-token\") pod \"2965df53-329d-4971-893e-3c60b72762c5\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " Oct 30 21:00:00.805414 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805343 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-workload-certs" pod "2965df53-329d-4971-893e-3c60b72762c5" (UID: "2965df53-329d-4971-893e-3c60b72762c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:00.805414 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805345 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-credential-socket" pod "4e15faa4-f00e-4316-9b78-fe0b40a09ac2" (UID: "4e15faa4-f00e-4316-9b78-fe0b40a09ac2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:00.805414 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805355 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2965df53-329d-4971-893e-3c60b72762c5-kube-api-access-jvp2j\" (UniqueName: \"kubernetes.io/projected/2965df53-329d-4971-893e-3c60b72762c5-kube-api-access-jvp2j\") pod \"2965df53-329d-4971-893e-3c60b72762c5\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " Oct 30 21:00:00.805737 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805457 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-envoy\") pod \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " Oct 30 21:00:00.805737 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805476 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2965df53-329d-4971-893e-3c60b72762c5-istiod-ca-cert" pod "2965df53-329d-4971-893e-3c60b72762c5" (UID: "2965df53-329d-4971-893e-3c60b72762c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:00.805737 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805486 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-workload-socket" pod "4e15faa4-f00e-4316-9b78-fe0b40a09ac2" (UID: "4e15faa4-f00e-4316-9b78-fe0b40a09ac2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:00.805737 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805504 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-data" pod "4e15faa4-f00e-4316-9b78-fe0b40a09ac2" (UID: "4e15faa4-f00e-4316-9b78-fe0b40a09ac2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:00.805737 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805529 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istiod-ca-cert\") pod \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " Oct 30 21:00:00.805737 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805576 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-workload-socket\") pod \"2965df53-329d-4971-893e-3c60b72762c5\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " Oct 30 21:00:00.805737 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805624 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2965df53-329d-4971-893e-3c60b72762c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2965df53-329d-4971-893e-3c60b72762c5-istio-podinfo\") pod \"2965df53-329d-4971-893e-3c60b72762c5\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " Oct 30 21:00:00.805737 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805659 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-token\" (UniqueName: \"kubernetes.io/projected/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-token\") pod \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " Oct 30 21:00:00.805737 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805714 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-istio-data\") pod \"2965df53-329d-4971-893e-3c60b72762c5\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " Oct 30 21:00:00.805737 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805739 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-credential-socket\") pod \"2965df53-329d-4971-893e-3c60b72762c5\" (UID: \"2965df53-329d-4971-893e-3c60b72762c5\") " Oct 30 21:00:00.806463 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805784 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-kube-api-access-64jjc\" (UniqueName: \"kubernetes.io/projected/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-kube-api-access-64jjc\") pod \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\" (UID: \"4e15faa4-f00e-4316-9b78-fe0b40a09ac2\") " Oct 30 21:00:00.806463 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805891 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-workload-socket" pod "2965df53-329d-4971-893e-3c60b72762c5" (UID: "2965df53-329d-4971-893e-3c60b72762c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:00.806463 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.805927 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istiod-ca-cert" pod "4e15faa4-f00e-4316-9b78-fe0b40a09ac2" (UID: "4e15faa4-f00e-4316-9b78-fe0b40a09ac2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:00.806463 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.806313 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.806463 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.806328 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-credential-socket" pod "2965df53-329d-4971-893e-3c60b72762c5" (UID: "2965df53-329d-4971-893e-3c60b72762c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:00.806463 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.806337 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.806463 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.806369 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-istio-data" pod "2965df53-329d-4971-893e-3c60b72762c5" (UID: "2965df53-329d-4971-893e-3c60b72762c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:00.806463 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.806380 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.806463 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.806429 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2965df53-329d-4971-893e-3c60b72762c5-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.806463 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.806446 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.806972 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.806499 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.806972 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.806549 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.806972 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.806583 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.808197 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.808168 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2965df53-329d-4971-893e-3c60b72762c5-kube-api-access-jvp2j" pod "2965df53-329d-4971-893e-3c60b72762c5" (UID: "2965df53-329d-4971-893e-3c60b72762c5"). InnerVolumeSpecName "kube-api-access-jvp2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:00.808332 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.808225 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-podinfo" pod "4e15faa4-f00e-4316-9b78-fe0b40a09ac2" (UID: "4e15faa4-f00e-4316-9b78-fe0b40a09ac2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:00.808461 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.808437 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-envoy" pod "4e15faa4-f00e-4316-9b78-fe0b40a09ac2" (UID: "4e15faa4-f00e-4316-9b78-fe0b40a09ac2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:00.808716 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.808696 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-kube-api-access-64jjc" pod "4e15faa4-f00e-4316-9b78-fe0b40a09ac2" (UID: "4e15faa4-f00e-4316-9b78-fe0b40a09ac2"). InnerVolumeSpecName "kube-api-access-64jjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:00.808772 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.808723 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2965df53-329d-4971-893e-3c60b72762c5-istio-token" pod "2965df53-329d-4971-893e-3c60b72762c5" (UID: "2965df53-329d-4971-893e-3c60b72762c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:00.808849 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.808827 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-token" pod "4e15faa4-f00e-4316-9b78-fe0b40a09ac2" (UID: "4e15faa4-f00e-4316-9b78-fe0b40a09ac2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:00.808910 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.808889 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-istio-envoy" pod "2965df53-329d-4971-893e-3c60b72762c5" (UID: "2965df53-329d-4971-893e-3c60b72762c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:00.809564 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.809548 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2965df53-329d-4971-893e-3c60b72762c5-istio-podinfo" pod "2965df53-329d-4971-893e-3c60b72762c5" (UID: "2965df53-329d-4971-893e-3c60b72762c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:00.888641 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.888602 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv"] Oct 30 21:00:00.907457 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.907427 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.907457 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.907453 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.907650 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.907468 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2965df53-329d-4971-893e-3c60b72762c5-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.907650 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.907480 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvp2j\" (UniqueName: \"kubernetes.io/projected/2965df53-329d-4971-893e-3c60b72762c5-kube-api-access-jvp2j\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.907650 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.907493 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.907650 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.907505 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2965df53-329d-4971-893e-3c60b72762c5-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.907650 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.907517 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.907650 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.907530 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.907650 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.907542 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2965df53-329d-4971-893e-3c60b72762c5-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.907650 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.907555 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64jjc\" (UniqueName: \"kubernetes.io/projected/4e15faa4-f00e-4316-9b78-fe0b40a09ac2-kube-api-access-64jjc\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:00.920015 ip-10-0-139-195 kubenswrapper[5755]: W1030 21:00:00.919982 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf12be6a_d0d3_47f5_bce7_65439f599fe6.slice/crio-9c0a81f9277f6c15e753a01bdd360d0275fb4a2b4893fad84ea537e75b7b17ed WatchSource:0}: Error finding container 9c0a81f9277f6c15e753a01bdd360d0275fb4a2b4893fad84ea537e75b7b17ed: Status 404 returned error can't find the container with id 9c0a81f9277f6c15e753a01bdd360d0275fb4a2b4893fad84ea537e75b7b17ed Oct 30 21:00:00.936960 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:00.936934 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj"] Oct 30 21:00:00.938134 ip-10-0-139-195 kubenswrapper[5755]: W1030 21:00:00.938105 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41470888_65ed_44de_9bcc_210cf168dece.slice/crio-c0fcb6e2cb9c0613c69b0c3694ad49036d3bba91aa19c5de91511469d629df0b WatchSource:0}: Error finding container c0fcb6e2cb9c0613c69b0c3694ad49036d3bba91aa19c5de91511469d629df0b: Status 404 returned error can't find the container with id c0fcb6e2cb9c0613c69b0c3694ad49036d3bba91aa19c5de91511469d629df0b Oct 30 21:00:01.203739 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:01.203705 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7"] Oct 30 21:00:01.212724 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:01.212700 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lm9p7"] Oct 30 21:00:01.320583 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:01.320558 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252"] Oct 30 21:00:01.325197 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:01.325176 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vk252"] Oct 30 21:00:01.784466 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:01.784424 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" event={"ID":"af12be6a-d0d3-47f5-bce7-65439f599fe6","Type":"ContainerStarted","Data":"489ed17ae44b715c2516db3d85c7556cd37f8b03ced88db5efd393b06b98d8f0"} Oct 30 21:00:01.784933 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:01.784524 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" event={"ID":"af12be6a-d0d3-47f5-bce7-65439f599fe6","Type":"ContainerStarted","Data":"9c0a81f9277f6c15e753a01bdd360d0275fb4a2b4893fad84ea537e75b7b17ed"} Oct 30 21:00:01.787029 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:01.787002 5755 generic.go:356] "Generic (PLEG): container finished" podID="41470888-65ed-44de-9bcc-210cf168dece" containerID="76daa4fc838372c6ea09a75fb0e895849c96cf3ff410a9c709f7dc2d117153c6" exitCode=60 Oct 30 21:00:01.787130 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:01.787045 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" event={"ID":"41470888-65ed-44de-9bcc-210cf168dece","Type":"ContainerDied","Data":"76daa4fc838372c6ea09a75fb0e895849c96cf3ff410a9c709f7dc2d117153c6"} Oct 30 21:00:01.787130 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:01.787083 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" event={"ID":"41470888-65ed-44de-9bcc-210cf168dece","Type":"ContainerStarted","Data":"c0fcb6e2cb9c0613c69b0c3694ad49036d3bba91aa19c5de91511469d629df0b"} Oct 30 21:00:01.787274 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:01.787263 5755 scope.go:122] "RemoveContainer" containerID="76daa4fc838372c6ea09a75fb0e895849c96cf3ff410a9c709f7dc2d117153c6" Oct 30 21:00:01.899706 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:01.899645 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" podStartSLOduration=1.899628871 podStartE2EDuration="1.899628871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:01.89900177 +0000 UTC m=+1964.825558710" watchObservedRunningTime="2026-10-30 21:00:01.899628871 +0000 UTC m=+1964.826185791" Oct 30 21:00:02.086969 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.086944 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9"] Oct 30 21:00:02.087147 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.087128 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" podUID="4d4872bc-be2d-4689-87aa-702ab0659e8c" containerName="discovery" containerID="cri-o://151e133f3ca7308bfd192957ce8f0dfe55678f0e62236741fec812673895112f" gracePeriod=30 Oct 30 21:00:02.209680 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.209639 5755 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vmrs9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:8080/ready\": dial tcp 10.134.0.101:8080: connect: connection refused" start-of-body= Oct 30 21:00:02.209796 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.209718 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" podUID="4d4872bc-be2d-4689-87aa-702ab0659e8c" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.101:8080/ready\": dial tcp 10.134.0.101:8080: connect: connection refused" Oct 30 21:00:02.332063 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.332038 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 21:00:02.423040 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.423013 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-kubeconfig\") pod \"4d4872bc-be2d-4689-87aa-702ab0659e8c\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " Oct 30 21:00:02.423220 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.423060 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4872bc-be2d-4689-87aa-702ab0659e8c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4872bc-be2d-4689-87aa-702ab0659e8c-local-certs\") pod \"4d4872bc-be2d-4689-87aa-702ab0659e8c\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " Oct 30 21:00:02.423220 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.423109 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d4872bc-be2d-4689-87aa-702ab0659e8c-kube-api-access-7mh8b\" (UniqueName: \"kubernetes.io/projected/4d4872bc-be2d-4689-87aa-702ab0659e8c-kube-api-access-7mh8b\") pod \"4d4872bc-be2d-4689-87aa-702ab0659e8c\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " Oct 30 21:00:02.423220 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.423130 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-cacerts\" (UniqueName: \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-cacerts\") pod \"4d4872bc-be2d-4689-87aa-702ab0659e8c\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " Oct 30 21:00:02.423220 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.423154 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-csr-dns-cert\") pod \"4d4872bc-be2d-4689-87aa-702ab0659e8c\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " Oct 30 21:00:02.423473 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.423228 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-csr-ca-configmap\") pod \"4d4872bc-be2d-4689-87aa-702ab0659e8c\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " Oct 30 21:00:02.423473 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.423314 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-token\" (UniqueName: \"kubernetes.io/projected/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-token\") pod \"4d4872bc-be2d-4689-87aa-702ab0659e8c\" (UID: \"4d4872bc-be2d-4689-87aa-702ab0659e8c\") " Oct 30 21:00:02.423735 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.423701 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-csr-ca-configmap" pod "4d4872bc-be2d-4689-87aa-702ab0659e8c" (UID: "4d4872bc-be2d-4689-87aa-702ab0659e8c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:02.425788 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.425749 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4872bc-be2d-4689-87aa-702ab0659e8c-local-certs" pod "4d4872bc-be2d-4689-87aa-702ab0659e8c" (UID: "4d4872bc-be2d-4689-87aa-702ab0659e8c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:02.425942 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.425796 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4872bc-be2d-4689-87aa-702ab0659e8c-kube-api-access-7mh8b" pod "4d4872bc-be2d-4689-87aa-702ab0659e8c" (UID: "4d4872bc-be2d-4689-87aa-702ab0659e8c"). InnerVolumeSpecName "kube-api-access-7mh8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:02.426040 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.426021 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-kubeconfig" pod "4d4872bc-be2d-4689-87aa-702ab0659e8c" (UID: "4d4872bc-be2d-4689-87aa-702ab0659e8c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:02.426097 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.426043 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-cacerts" pod "4d4872bc-be2d-4689-87aa-702ab0659e8c" (UID: "4d4872bc-be2d-4689-87aa-702ab0659e8c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:02.426261 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.426237 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-token" pod "4d4872bc-be2d-4689-87aa-702ab0659e8c" (UID: "4d4872bc-be2d-4689-87aa-702ab0659e8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:02.426347 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.426331 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-csr-dns-cert" pod "4d4872bc-be2d-4689-87aa-702ab0659e8c" (UID: "4d4872bc-be2d-4689-87aa-702ab0659e8c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:02.524372 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.524302 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:02.524372 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.524326 5755 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-kubeconfig\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:02.524372 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.524336 5755 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4872bc-be2d-4689-87aa-702ab0659e8c-local-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:02.524372 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.524349 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mh8b\" (UniqueName: \"kubernetes.io/projected/4d4872bc-be2d-4689-87aa-702ab0659e8c-kube-api-access-7mh8b\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:02.524372 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.524358 5755 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-cacerts\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:02.524372 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.524366 5755 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-csr-dns-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:02.524372 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.524374 5755 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4d4872bc-be2d-4689-87aa-702ab0659e8c-istio-csr-ca-configmap\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:02.795190 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.795114 5755 generic.go:356] "Generic (PLEG): container finished" podID="4d4872bc-be2d-4689-87aa-702ab0659e8c" containerID="151e133f3ca7308bfd192957ce8f0dfe55678f0e62236741fec812673895112f" exitCode=0 Oct 30 21:00:02.795190 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.795162 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" Oct 30 21:00:02.795741 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.795192 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" event={"ID":"4d4872bc-be2d-4689-87aa-702ab0659e8c","Type":"ContainerDied","Data":"151e133f3ca7308bfd192957ce8f0dfe55678f0e62236741fec812673895112f"} Oct 30 21:00:02.795741 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.795243 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9" event={"ID":"4d4872bc-be2d-4689-87aa-702ab0659e8c","Type":"ContainerDied","Data":"90df6af32e290e84d01c806e3daed0152c35859cd83863513fca4e7b1fe0e98b"} Oct 30 21:00:02.795741 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.795263 5755 scope.go:122] "RemoveContainer" containerID="151e133f3ca7308bfd192957ce8f0dfe55678f0e62236741fec812673895112f" Oct 30 21:00:02.797120 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.797097 5755 generic.go:356] "Generic (PLEG): container finished" podID="af12be6a-d0d3-47f5-bce7-65439f599fe6" containerID="489ed17ae44b715c2516db3d85c7556cd37f8b03ced88db5efd393b06b98d8f0" exitCode=60 Oct 30 21:00:02.797226 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.797187 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" event={"ID":"af12be6a-d0d3-47f5-bce7-65439f599fe6","Type":"ContainerDied","Data":"489ed17ae44b715c2516db3d85c7556cd37f8b03ced88db5efd393b06b98d8f0"} Oct 30 21:00:02.797463 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.797440 5755 scope.go:122] "RemoveContainer" containerID="489ed17ae44b715c2516db3d85c7556cd37f8b03ced88db5efd393b06b98d8f0" Oct 30 21:00:02.799453 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.799434 5755 generic.go:356] "Generic (PLEG): container finished" podID="41470888-65ed-44de-9bcc-210cf168dece" containerID="7a637f00577070fe58599e673d6e076631ee96574e8322e1f39726368041ecdb" exitCode=60 Oct 30 21:00:02.799518 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.799465 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" event={"ID":"41470888-65ed-44de-9bcc-210cf168dece","Type":"ContainerDied","Data":"7a637f00577070fe58599e673d6e076631ee96574e8322e1f39726368041ecdb"} Oct 30 21:00:02.799774 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.799761 5755 scope.go:122] "RemoveContainer" containerID="7a637f00577070fe58599e673d6e076631ee96574e8322e1f39726368041ecdb" Oct 30 21:00:02.800774 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:02.800750 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj_odh-ai-gateway-infra(41470888-65ed-44de-9bcc-210cf168dece)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" podUID="41470888-65ed-44de-9bcc-210cf168dece" Oct 30 21:00:02.809890 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.809871 5755 scope.go:122] "RemoveContainer" containerID="151e133f3ca7308bfd192957ce8f0dfe55678f0e62236741fec812673895112f" Oct 30 21:00:02.810134 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:02.810116 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151e133f3ca7308bfd192957ce8f0dfe55678f0e62236741fec812673895112f\": container with ID starting with 151e133f3ca7308bfd192957ce8f0dfe55678f0e62236741fec812673895112f not found: ID does not exist" containerID="151e133f3ca7308bfd192957ce8f0dfe55678f0e62236741fec812673895112f" Oct 30 21:00:02.810215 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.810141 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151e133f3ca7308bfd192957ce8f0dfe55678f0e62236741fec812673895112f"} err="failed to get container status \"151e133f3ca7308bfd192957ce8f0dfe55678f0e62236741fec812673895112f\": rpc error: code = NotFound desc = could not find container \"151e133f3ca7308bfd192957ce8f0dfe55678f0e62236741fec812673895112f\": container with ID starting with 151e133f3ca7308bfd192957ce8f0dfe55678f0e62236741fec812673895112f not found: ID does not exist" Oct 30 21:00:02.810215 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:02.810157 5755 scope.go:122] "RemoveContainer" containerID="76daa4fc838372c6ea09a75fb0e895849c96cf3ff410a9c709f7dc2d117153c6" Oct 30 21:00:03.203838 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:03.203768 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9"] Oct 30 21:00:03.211598 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:03.211574 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2965df53-329d-4971-893e-3c60b72762c5" path="/var/lib/kubelet/pods/2965df53-329d-4971-893e-3c60b72762c5/volumes" Oct 30 21:00:03.212009 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:03.211997 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e15faa4-f00e-4316-9b78-fe0b40a09ac2" path="/var/lib/kubelet/pods/4e15faa4-f00e-4316-9b78-fe0b40a09ac2/volumes" Oct 30 21:00:03.219675 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:03.219646 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vmrs9"] Oct 30 21:00:03.807806 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:03.807772 5755 scope.go:122] "RemoveContainer" containerID="7a637f00577070fe58599e673d6e076631ee96574e8322e1f39726368041ecdb" Oct 30 21:00:03.808231 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:03.808014 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj_odh-ai-gateway-infra(41470888-65ed-44de-9bcc-210cf168dece)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" podUID="41470888-65ed-44de-9bcc-210cf168dece" Oct 30 21:00:03.810475 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:03.810446 5755 generic.go:356] "Generic (PLEG): container finished" podID="af12be6a-d0d3-47f5-bce7-65439f599fe6" containerID="ac3425be0aefe16d0517b7ad7e3e04b98bcbd7ccf8f3bca86209c9129749e66c" exitCode=60 Oct 30 21:00:03.810606 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:03.810502 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" event={"ID":"af12be6a-d0d3-47f5-bce7-65439f599fe6","Type":"ContainerDied","Data":"ac3425be0aefe16d0517b7ad7e3e04b98bcbd7ccf8f3bca86209c9129749e66c"} Oct 30 21:00:03.810606 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:03.810558 5755 scope.go:122] "RemoveContainer" containerID="489ed17ae44b715c2516db3d85c7556cd37f8b03ced88db5efd393b06b98d8f0" Oct 30 21:00:03.810801 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:03.810783 5755 scope.go:122] "RemoveContainer" containerID="ac3425be0aefe16d0517b7ad7e3e04b98bcbd7ccf8f3bca86209c9129749e66c" Oct 30 21:00:03.811064 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:03.811043 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv_odh-ai-gateway-infra(af12be6a-d0d3-47f5-bce7-65439f599fe6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" podUID="af12be6a-d0d3-47f5-bce7-65439f599fe6" Oct 30 21:00:04.818739 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:04.818713 5755 scope.go:122] "RemoveContainer" containerID="ac3425be0aefe16d0517b7ad7e3e04b98bcbd7ccf8f3bca86209c9129749e66c" Oct 30 21:00:04.819116 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:04.818935 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv_odh-ai-gateway-infra(af12be6a-d0d3-47f5-bce7-65439f599fe6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" podUID="af12be6a-d0d3-47f5-bce7-65439f599fe6" Oct 30 21:00:05.211591 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:05.211501 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d4872bc-be2d-4689-87aa-702ab0659e8c" path="/var/lib/kubelet/pods/4d4872bc-be2d-4689-87aa-702ab0659e8c/volumes" Oct 30 21:00:06.100870 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.100832 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq"] Oct 30 21:00:06.101291 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.101039 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" podUID="50baec58-01fc-4d2a-914a-1baf6a6cda9c" containerName="maas-api" containerID="cri-o://b4a0b97151c1a3a605f8f322c34259da88fafe99ba7d95db27bbbd4c28e6357e" gracePeriod=30 Oct 30 21:00:06.386155 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.386129 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 21:00:06.463687 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.463655 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-maas-api-tls\") pod \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " Oct 30 21:00:06.463687 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.463690 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50baec58-01fc-4d2a-914a-1baf6a6cda9c-kube-api-access-pbjdx\" (UniqueName: \"kubernetes.io/projected/50baec58-01fc-4d2a-914a-1baf6a6cda9c-kube-api-access-pbjdx\") pod \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " Oct 30 21:00:06.463909 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.463746 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50baec58-01fc-4d2a-914a-1baf6a6cda9c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50baec58-01fc-4d2a-914a-1baf6a6cda9c-openshift-service-ca\") pod \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " Oct 30 21:00:06.463909 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.463829 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-metrics-certs\") pod \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\" (UID: \"50baec58-01fc-4d2a-914a-1baf6a6cda9c\") " Oct 30 21:00:06.464177 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.464149 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50baec58-01fc-4d2a-914a-1baf6a6cda9c-openshift-service-ca" pod "50baec58-01fc-4d2a-914a-1baf6a6cda9c" (UID: "50baec58-01fc-4d2a-914a-1baf6a6cda9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:06.466090 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.466053 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-maas-api-tls" pod "50baec58-01fc-4d2a-914a-1baf6a6cda9c" (UID: "50baec58-01fc-4d2a-914a-1baf6a6cda9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:06.466090 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.466069 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-metrics-certs" pod "50baec58-01fc-4d2a-914a-1baf6a6cda9c" (UID: "50baec58-01fc-4d2a-914a-1baf6a6cda9c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:06.466090 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.466075 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50baec58-01fc-4d2a-914a-1baf6a6cda9c-kube-api-access-pbjdx" pod "50baec58-01fc-4d2a-914a-1baf6a6cda9c" (UID: "50baec58-01fc-4d2a-914a-1baf6a6cda9c"). InnerVolumeSpecName "kube-api-access-pbjdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:06.565540 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.565504 5755 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-metrics-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:06.565540 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.565533 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50baec58-01fc-4d2a-914a-1baf6a6cda9c-maas-api-tls\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:06.565540 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.565544 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbjdx\" (UniqueName: \"kubernetes.io/projected/50baec58-01fc-4d2a-914a-1baf6a6cda9c-kube-api-access-pbjdx\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:06.565747 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.565555 5755 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50baec58-01fc-4d2a-914a-1baf6a6cda9c-openshift-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:06.832951 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.832917 5755 generic.go:356] "Generic (PLEG): container finished" podID="50baec58-01fc-4d2a-914a-1baf6a6cda9c" containerID="b4a0b97151c1a3a605f8f322c34259da88fafe99ba7d95db27bbbd4c28e6357e" exitCode=0 Oct 30 21:00:06.833103 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.832961 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" Oct 30 21:00:06.833103 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.832967 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" event={"ID":"50baec58-01fc-4d2a-914a-1baf6a6cda9c","Type":"ContainerDied","Data":"b4a0b97151c1a3a605f8f322c34259da88fafe99ba7d95db27bbbd4c28e6357e"} Oct 30 21:00:06.833103 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.833033 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq" event={"ID":"50baec58-01fc-4d2a-914a-1baf6a6cda9c","Type":"ContainerDied","Data":"7595405adbb3e3ef4a17d0d7b7cdbc6719e875537915fc277784931588cda3dc"} Oct 30 21:00:06.833103 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.833054 5755 scope.go:122] "RemoveContainer" containerID="b4a0b97151c1a3a605f8f322c34259da88fafe99ba7d95db27bbbd4c28e6357e" Oct 30 21:00:06.843845 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.843825 5755 scope.go:122] "RemoveContainer" containerID="b4a0b97151c1a3a605f8f322c34259da88fafe99ba7d95db27bbbd4c28e6357e" Oct 30 21:00:06.844184 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:06.844164 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a0b97151c1a3a605f8f322c34259da88fafe99ba7d95db27bbbd4c28e6357e\": container with ID starting with b4a0b97151c1a3a605f8f322c34259da88fafe99ba7d95db27bbbd4c28e6357e not found: ID does not exist" containerID="b4a0b97151c1a3a605f8f322c34259da88fafe99ba7d95db27bbbd4c28e6357e" Oct 30 21:00:06.844225 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.844192 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a0b97151c1a3a605f8f322c34259da88fafe99ba7d95db27bbbd4c28e6357e"} err="failed to get container status \"b4a0b97151c1a3a605f8f322c34259da88fafe99ba7d95db27bbbd4c28e6357e\": rpc error: code = NotFound desc = could not find container \"b4a0b97151c1a3a605f8f322c34259da88fafe99ba7d95db27bbbd4c28e6357e\": container with ID starting with b4a0b97151c1a3a605f8f322c34259da88fafe99ba7d95db27bbbd4c28e6357e not found: ID does not exist" Oct 30 21:00:06.999285 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:06.999246 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq"] Oct 30 21:00:07.003638 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.003612 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-386743-6b947fc95c-vcctq"] Oct 30 21:00:07.211144 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.211063 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50baec58-01fc-4d2a-914a-1baf6a6cda9c" path="/var/lib/kubelet/pods/50baec58-01fc-4d2a-914a-1baf6a6cda9c/volumes" Oct 30 21:00:07.446736 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.446688 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv"] Oct 30 21:00:07.447158 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.447143 5755 cpu_manager.go:414] "containerMap: removing container" podUID="4d4872bc-be2d-4689-87aa-702ab0659e8c" containerName="discovery" Oct 30 21:00:07.447201 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.447161 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d4872bc-be2d-4689-87aa-702ab0659e8c" containerName="discovery" Oct 30 21:00:07.447201 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.447174 5755 cpu_manager.go:414] "containerMap: removing container" podUID="50baec58-01fc-4d2a-914a-1baf6a6cda9c" containerName="maas-api" Oct 30 21:00:07.447201 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.447180 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50baec58-01fc-4d2a-914a-1baf6a6cda9c" containerName="maas-api" Oct 30 21:00:07.447311 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.447202 5755 cpu_manager.go:414] "containerMap: removing container" podUID="4e15faa4-f00e-4316-9b78-fe0b40a09ac2" containerName="istio-proxy" Oct 30 21:00:07.447311 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.447208 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e15faa4-f00e-4316-9b78-fe0b40a09ac2" containerName="istio-proxy" Oct 30 21:00:07.447311 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.447217 5755 cpu_manager.go:414] "containerMap: removing container" podUID="2965df53-329d-4971-893e-3c60b72762c5" containerName="istio-proxy" Oct 30 21:00:07.447311 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.447222 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2965df53-329d-4971-893e-3c60b72762c5" containerName="istio-proxy" Oct 30 21:00:07.447311 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.447282 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d4872bc-be2d-4689-87aa-702ab0659e8c" containerName="discovery" Oct 30 21:00:07.447311 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.447293 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="50baec58-01fc-4d2a-914a-1baf6a6cda9c" containerName="maas-api" Oct 30 21:00:07.447311 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.447301 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e15faa4-f00e-4316-9b78-fe0b40a09ac2" containerName="istio-proxy" Oct 30 21:00:07.447311 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.447309 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="2965df53-329d-4971-893e-3c60b72762c5" containerName="istio-proxy" Oct 30 21:00:07.450339 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.450324 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" Oct 30 21:00:07.475692 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.475667 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16840665-c3f7-48bf-a2bc-695c0be23df0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4950931-w7fjv\" (UID: \"16840665-c3f7-48bf-a2bc-695c0be23df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" Oct 30 21:00:07.475828 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.475809 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lm8b\" (UniqueName: \"kubernetes.io/projected/16840665-c3f7-48bf-a2bc-695c0be23df0-kube-api-access-4lm8b\") pod \"maas-api-revoke-sub-a4950931-w7fjv\" (UID: \"16840665-c3f7-48bf-a2bc-695c0be23df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" Oct 30 21:00:07.475884 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.475865 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16840665-c3f7-48bf-a2bc-695c0be23df0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4950931-w7fjv\" (UID: \"16840665-c3f7-48bf-a2bc-695c0be23df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" Oct 30 21:00:07.508229 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.508149 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv"] Oct 30 21:00:07.552214 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.552187 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:00:07.576898 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.576869 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16840665-c3f7-48bf-a2bc-695c0be23df0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4950931-w7fjv\" (UID: \"16840665-c3f7-48bf-a2bc-695c0be23df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" Oct 30 21:00:07.577028 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.576959 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lm8b\" (UniqueName: \"kubernetes.io/projected/16840665-c3f7-48bf-a2bc-695c0be23df0-kube-api-access-4lm8b\") pod \"maas-api-revoke-sub-a4950931-w7fjv\" (UID: \"16840665-c3f7-48bf-a2bc-695c0be23df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" Oct 30 21:00:07.577028 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.576989 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16840665-c3f7-48bf-a2bc-695c0be23df0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4950931-w7fjv\" (UID: \"16840665-c3f7-48bf-a2bc-695c0be23df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" Oct 30 21:00:07.577516 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.577498 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16840665-c3f7-48bf-a2bc-695c0be23df0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4950931-w7fjv\" (UID: \"16840665-c3f7-48bf-a2bc-695c0be23df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" Oct 30 21:00:07.579228 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.579212 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16840665-c3f7-48bf-a2bc-695c0be23df0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4950931-w7fjv\" (UID: \"16840665-c3f7-48bf-a2bc-695c0be23df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" Oct 30 21:00:07.634788 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.634764 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lm8b\" (UniqueName: \"kubernetes.io/projected/16840665-c3f7-48bf-a2bc-695c0be23df0-kube-api-access-4lm8b\") pod \"maas-api-revoke-sub-a4950931-w7fjv\" (UID: \"16840665-c3f7-48bf-a2bc-695c0be23df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" Oct 30 21:00:07.765511 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:07.765439 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" Oct 30 21:00:08.144374 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:08.144341 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv"] Oct 30 21:00:08.145346 ip-10-0-139-195 kubenswrapper[5755]: W1030 21:00:08.145315 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16840665_c3f7_48bf_a2bc_695c0be23df0.slice/crio-ae4a5fdfb88589439a2f20e9ef7240f73663fd9c1a943d0024ab5054dae9e23a WatchSource:0}: Error finding container ae4a5fdfb88589439a2f20e9ef7240f73663fd9c1a943d0024ab5054dae9e23a: Status 404 returned error can't find the container with id ae4a5fdfb88589439a2f20e9ef7240f73663fd9c1a943d0024ab5054dae9e23a Oct 30 21:00:08.318447 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:08.318414 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj"] Oct 30 21:00:08.324803 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:08.324778 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-386743-mpplj"] Oct 30 21:00:08.853886 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:08.853786 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" event={"ID":"16840665-c3f7-48bf-a2bc-695c0be23df0","Type":"ContainerStarted","Data":"63613f25fb4846b5557dbc939e0e0322f25837b2ee2c1e63391d1d7072066d88"} Oct 30 21:00:08.853886 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:08.853845 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" event={"ID":"16840665-c3f7-48bf-a2bc-695c0be23df0","Type":"ContainerStarted","Data":"ae4a5fdfb88589439a2f20e9ef7240f73663fd9c1a943d0024ab5054dae9e23a"} Oct 30 21:00:08.970543 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:08.970487 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" podStartSLOduration=1.970471842 podStartE2EDuration="1.970471842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:08.96759112 +0000 UTC m=+1971.894148055" watchObservedRunningTime="2026-10-30 21:00:08.970471842 +0000 UTC m=+1971.897028806" Oct 30 21:00:09.211524 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:09.211454 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2d186b-4be6-4e85-9614-495004fa6ab2" path="/var/lib/kubelet/pods/ec2d186b-4be6-4e85-9614-495004fa6ab2/volumes" Oct 30 21:00:09.861812 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:09.861778 5755 generic.go:356] "Generic (PLEG): container finished" podID="16840665-c3f7-48bf-a2bc-695c0be23df0" containerID="63613f25fb4846b5557dbc939e0e0322f25837b2ee2c1e63391d1d7072066d88" exitCode=0 Oct 30 21:00:09.862296 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:09.861857 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" event={"ID":"16840665-c3f7-48bf-a2bc-695c0be23df0","Type":"ContainerDied","Data":"63613f25fb4846b5557dbc939e0e0322f25837b2ee2c1e63391d1d7072066d88"} Oct 30 21:00:11.010984 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:11.010961 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" Oct 30 21:00:11.115213 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:11.115180 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16840665-c3f7-48bf-a2bc-695c0be23df0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16840665-c3f7-48bf-a2bc-695c0be23df0-maas-api-xks-ca\") pod \"16840665-c3f7-48bf-a2bc-695c0be23df0\" (UID: \"16840665-c3f7-48bf-a2bc-695c0be23df0\") " Oct 30 21:00:11.115417 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:11.115251 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16840665-c3f7-48bf-a2bc-695c0be23df0-kube-api-access-4lm8b\" (UniqueName: \"kubernetes.io/projected/16840665-c3f7-48bf-a2bc-695c0be23df0-kube-api-access-4lm8b\") pod \"16840665-c3f7-48bf-a2bc-695c0be23df0\" (UID: \"16840665-c3f7-48bf-a2bc-695c0be23df0\") " Oct 30 21:00:11.115417 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:11.115312 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16840665-c3f7-48bf-a2bc-695c0be23df0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16840665-c3f7-48bf-a2bc-695c0be23df0-maas-api-openshift-ca\") pod \"16840665-c3f7-48bf-a2bc-695c0be23df0\" (UID: \"16840665-c3f7-48bf-a2bc-695c0be23df0\") " Oct 30 21:00:11.115719 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:11.115689 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16840665-c3f7-48bf-a2bc-695c0be23df0-maas-api-openshift-ca" pod "16840665-c3f7-48bf-a2bc-695c0be23df0" (UID: "16840665-c3f7-48bf-a2bc-695c0be23df0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:11.117479 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:11.117456 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16840665-c3f7-48bf-a2bc-695c0be23df0-maas-api-xks-ca" pod "16840665-c3f7-48bf-a2bc-695c0be23df0" (UID: "16840665-c3f7-48bf-a2bc-695c0be23df0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:11.117566 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:11.117505 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16840665-c3f7-48bf-a2bc-695c0be23df0-kube-api-access-4lm8b" pod "16840665-c3f7-48bf-a2bc-695c0be23df0" (UID: "16840665-c3f7-48bf-a2bc-695c0be23df0"). InnerVolumeSpecName "kube-api-access-4lm8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:11.216492 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:11.216454 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16840665-c3f7-48bf-a2bc-695c0be23df0-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.216492 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:11.216485 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lm8b\" (UniqueName: \"kubernetes.io/projected/16840665-c3f7-48bf-a2bc-695c0be23df0-kube-api-access-4lm8b\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.216492 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:11.216499 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16840665-c3f7-48bf-a2bc-695c0be23df0-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.260216 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:11.260172 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:00:11.881770 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:11.881725 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" event={"ID":"16840665-c3f7-48bf-a2bc-695c0be23df0","Type":"ContainerDied","Data":"ae4a5fdfb88589439a2f20e9ef7240f73663fd9c1a943d0024ab5054dae9e23a"} Oct 30 21:00:11.882011 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:11.881785 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv" Oct 30 21:00:11.882011 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:11.881799 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae4a5fdfb88589439a2f20e9ef7240f73663fd9c1a943d0024ab5054dae9e23a" Oct 30 21:00:14.549648 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.549596 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" containerID="cri-o://9277f31e4a8bf2717b0350b4c0eff63deadbc52afb5c345df700bbf6c7571fb6" gracePeriod=45 Oct 30 21:00:14.598094 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.598061 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" containerID="cri-o://99214e5c1a44b99a33692e72745e9537685c5e4d8ea90c51613b8e908ec6f8bb" gracePeriod=45 Oct 30 21:00:14.599916 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.599886 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" containerID="cri-o://4368d885a4d2e5ac0f5bb4a7ea6f14a2ab4962f9682d3ccd8d17d818a14eb91d" gracePeriod=45 Oct 30 21:00:14.862083 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.862058 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 21:00:14.895770 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.895748 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 21:00:14.898790 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.898773 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 21:00:14.901571 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.901548 5755 generic.go:356] "Generic (PLEG): container finished" podID="aa094251-9344-47b6-bbcf-79823b693371" containerID="4368d885a4d2e5ac0f5bb4a7ea6f14a2ab4962f9682d3ccd8d17d818a14eb91d" exitCode=0 Oct 30 21:00:14.901682 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.901579 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" Oct 30 21:00:14.901682 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.901580 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" event={"ID":"aa094251-9344-47b6-bbcf-79823b693371","Type":"ContainerDied","Data":"4368d885a4d2e5ac0f5bb4a7ea6f14a2ab4962f9682d3ccd8d17d818a14eb91d"} Oct 30 21:00:14.901682 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.901617 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f" event={"ID":"aa094251-9344-47b6-bbcf-79823b693371","Type":"ContainerDied","Data":"ea166dc46deb35a43ece39119583ecd16caf41f8c2c6f404fe6aab840490d16b"} Oct 30 21:00:14.901682 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.901643 5755 scope.go:122] "RemoveContainer" containerID="4368d885a4d2e5ac0f5bb4a7ea6f14a2ab4962f9682d3ccd8d17d818a14eb91d" Oct 30 21:00:14.902988 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.902968 5755 generic.go:356] "Generic (PLEG): container finished" podID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerID="99214e5c1a44b99a33692e72745e9537685c5e4d8ea90c51613b8e908ec6f8bb" exitCode=0 Oct 30 21:00:14.903069 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.903001 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" Oct 30 21:00:14.903069 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.903044 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" event={"ID":"d16f785a-baaa-466b-9965-51cf8a8a0bfe","Type":"ContainerDied","Data":"99214e5c1a44b99a33692e72745e9537685c5e4d8ea90c51613b8e908ec6f8bb"} Oct 30 21:00:14.903152 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.903078 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445" event={"ID":"d16f785a-baaa-466b-9965-51cf8a8a0bfe","Type":"ContainerDied","Data":"e23d293cf3d0d51d115b2072279445f3dfdf17cde41323132ec13d772255920e"} Oct 30 21:00:14.904651 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.904628 5755 generic.go:356] "Generic (PLEG): container finished" podID="440b1853-8437-4777-9b63-afddbbb3f89a" containerID="9277f31e4a8bf2717b0350b4c0eff63deadbc52afb5c345df700bbf6c7571fb6" exitCode=0 Oct 30 21:00:14.904745 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.904667 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" Oct 30 21:00:14.904745 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.904713 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" event={"ID":"440b1853-8437-4777-9b63-afddbbb3f89a","Type":"ContainerDied","Data":"9277f31e4a8bf2717b0350b4c0eff63deadbc52afb5c345df700bbf6c7571fb6"} Oct 30 21:00:14.904844 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.904746 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9" event={"ID":"440b1853-8437-4777-9b63-afddbbb3f89a","Type":"ContainerDied","Data":"b48e5a8901f0eb10da1feccb8cf9b2624929468dfb1ba06b5d8b5140b310c981"} Oct 30 21:00:14.911230 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.911212 5755 scope.go:122] "RemoveContainer" containerID="4368d885a4d2e5ac0f5bb4a7ea6f14a2ab4962f9682d3ccd8d17d818a14eb91d" Oct 30 21:00:14.911523 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:14.911504 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4368d885a4d2e5ac0f5bb4a7ea6f14a2ab4962f9682d3ccd8d17d818a14eb91d\": container with ID starting with 4368d885a4d2e5ac0f5bb4a7ea6f14a2ab4962f9682d3ccd8d17d818a14eb91d not found: ID does not exist" containerID="4368d885a4d2e5ac0f5bb4a7ea6f14a2ab4962f9682d3ccd8d17d818a14eb91d" Oct 30 21:00:14.911599 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.911532 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4368d885a4d2e5ac0f5bb4a7ea6f14a2ab4962f9682d3ccd8d17d818a14eb91d"} err="failed to get container status \"4368d885a4d2e5ac0f5bb4a7ea6f14a2ab4962f9682d3ccd8d17d818a14eb91d\": rpc error: code = NotFound desc = could not find container \"4368d885a4d2e5ac0f5bb4a7ea6f14a2ab4962f9682d3ccd8d17d818a14eb91d\": container with ID starting with 4368d885a4d2e5ac0f5bb4a7ea6f14a2ab4962f9682d3ccd8d17d818a14eb91d not found: ID does not exist" Oct 30 21:00:14.911599 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.911549 5755 scope.go:122] "RemoveContainer" containerID="99214e5c1a44b99a33692e72745e9537685c5e4d8ea90c51613b8e908ec6f8bb" Oct 30 21:00:14.920244 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.920227 5755 scope.go:122] "RemoveContainer" containerID="99214e5c1a44b99a33692e72745e9537685c5e4d8ea90c51613b8e908ec6f8bb" Oct 30 21:00:14.920515 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:14.920495 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99214e5c1a44b99a33692e72745e9537685c5e4d8ea90c51613b8e908ec6f8bb\": container with ID starting with 99214e5c1a44b99a33692e72745e9537685c5e4d8ea90c51613b8e908ec6f8bb not found: ID does not exist" containerID="99214e5c1a44b99a33692e72745e9537685c5e4d8ea90c51613b8e908ec6f8bb" Oct 30 21:00:14.920583 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.920523 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99214e5c1a44b99a33692e72745e9537685c5e4d8ea90c51613b8e908ec6f8bb"} err="failed to get container status \"99214e5c1a44b99a33692e72745e9537685c5e4d8ea90c51613b8e908ec6f8bb\": rpc error: code = NotFound desc = could not find container \"99214e5c1a44b99a33692e72745e9537685c5e4d8ea90c51613b8e908ec6f8bb\": container with ID starting with 99214e5c1a44b99a33692e72745e9537685c5e4d8ea90c51613b8e908ec6f8bb not found: ID does not exist" Oct 30 21:00:14.920583 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.920539 5755 scope.go:122] "RemoveContainer" containerID="9277f31e4a8bf2717b0350b4c0eff63deadbc52afb5c345df700bbf6c7571fb6" Oct 30 21:00:14.932692 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.932674 5755 scope.go:122] "RemoveContainer" containerID="9277f31e4a8bf2717b0350b4c0eff63deadbc52afb5c345df700bbf6c7571fb6" Oct 30 21:00:14.932966 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:14.932945 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9277f31e4a8bf2717b0350b4c0eff63deadbc52afb5c345df700bbf6c7571fb6\": container with ID starting with 9277f31e4a8bf2717b0350b4c0eff63deadbc52afb5c345df700bbf6c7571fb6 not found: ID does not exist" containerID="9277f31e4a8bf2717b0350b4c0eff63deadbc52afb5c345df700bbf6c7571fb6" Oct 30 21:00:14.933036 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.932977 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9277f31e4a8bf2717b0350b4c0eff63deadbc52afb5c345df700bbf6c7571fb6"} err="failed to get container status \"9277f31e4a8bf2717b0350b4c0eff63deadbc52afb5c345df700bbf6c7571fb6\": rpc error: code = NotFound desc = could not find container \"9277f31e4a8bf2717b0350b4c0eff63deadbc52afb5c345df700bbf6c7571fb6\": container with ID starting with 9277f31e4a8bf2717b0350b4c0eff63deadbc52afb5c345df700bbf6c7571fb6 not found: ID does not exist" Oct 30 21:00:14.951297 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951270 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-tmp-dir\") pod \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " Oct 30 21:00:14.951424 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951306 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-model-cache\") pod \"aa094251-9344-47b6-bbcf-79823b693371\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " Oct 30 21:00:14.951424 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951324 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-tmp-dir\") pod \"440b1853-8437-4777-9b63-afddbbb3f89a\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " Oct 30 21:00:14.951523 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951422 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa094251-9344-47b6-bbcf-79823b693371-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa094251-9344-47b6-bbcf-79823b693371-tls-certs\") pod \"aa094251-9344-47b6-bbcf-79823b693371\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " Oct 30 21:00:14.951523 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951455 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-dshm\") pod \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " Oct 30 21:00:14.951523 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951483 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-home\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-home\") pod \"aa094251-9344-47b6-bbcf-79823b693371\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " Oct 30 21:00:14.951685 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951530 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440b1853-8437-4777-9b63-afddbbb3f89a-tls-certs\" (UniqueName: \"kubernetes.io/secret/440b1853-8437-4777-9b63-afddbbb3f89a-tls-certs\") pod \"440b1853-8437-4777-9b63-afddbbb3f89a\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " Oct 30 21:00:14.951685 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951574 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d16f785a-baaa-466b-9965-51cf8a8a0bfe-tls-certs\" (UniqueName: \"kubernetes.io/secret/d16f785a-baaa-466b-9965-51cf8a8a0bfe-tls-certs\") pod \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " Oct 30 21:00:14.951685 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951568 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-tmp-dir" pod "d16f785a-baaa-466b-9965-51cf8a8a0bfe" (UID: "d16f785a-baaa-466b-9965-51cf8a8a0bfe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:14.951685 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951615 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440b1853-8437-4777-9b63-afddbbb3f89a-kube-api-access-t82bw\" (UniqueName: \"kubernetes.io/projected/440b1853-8437-4777-9b63-afddbbb3f89a-kube-api-access-t82bw\") pod \"440b1853-8437-4777-9b63-afddbbb3f89a\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " Oct 30 21:00:14.951685 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951616 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-tmp-dir" pod "440b1853-8437-4777-9b63-afddbbb3f89a" (UID: "440b1853-8437-4777-9b63-afddbbb3f89a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:14.951685 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951644 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa094251-9344-47b6-bbcf-79823b693371-kube-api-access-dxkx5\" (UniqueName: \"kubernetes.io/projected/aa094251-9344-47b6-bbcf-79823b693371-kube-api-access-dxkx5\") pod \"aa094251-9344-47b6-bbcf-79823b693371\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " Oct 30 21:00:14.951993 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951695 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-model-cache\") pod \"440b1853-8437-4777-9b63-afddbbb3f89a\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " Oct 30 21:00:14.951993 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951744 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-dshm\") pod \"440b1853-8437-4777-9b63-afddbbb3f89a\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " Oct 30 21:00:14.951993 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951773 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-dshm\") pod \"aa094251-9344-47b6-bbcf-79823b693371\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " Oct 30 21:00:14.951993 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951788 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-home" pod "aa094251-9344-47b6-bbcf-79823b693371" (UID: "aa094251-9344-47b6-bbcf-79823b693371"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:14.951993 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951823 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-model-cache\") pod \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " Oct 30 21:00:14.951993 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951853 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16f785a-baaa-466b-9965-51cf8a8a0bfe-kube-api-access-bqshs\" (UniqueName: \"kubernetes.io/projected/d16f785a-baaa-466b-9965-51cf8a8a0bfe-kube-api-access-bqshs\") pod \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " Oct 30 21:00:14.951993 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951886 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-tmp-dir\") pod \"aa094251-9344-47b6-bbcf-79823b693371\" (UID: \"aa094251-9344-47b6-bbcf-79823b693371\") " Oct 30 21:00:14.951993 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951916 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-home\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-home\") pod \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\" (UID: \"d16f785a-baaa-466b-9965-51cf8a8a0bfe\") " Oct 30 21:00:14.951993 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951971 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-home\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-home\") pod \"440b1853-8437-4777-9b63-afddbbb3f89a\" (UID: \"440b1853-8437-4777-9b63-afddbbb3f89a\") " Oct 30 21:00:14.952520 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951887 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-model-cache" pod "aa094251-9344-47b6-bbcf-79823b693371" (UID: "aa094251-9344-47b6-bbcf-79823b693371"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:14.952520 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.951972 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-model-cache" pod "440b1853-8437-4777-9b63-afddbbb3f89a" (UID: "440b1853-8437-4777-9b63-afddbbb3f89a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:14.952670 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.952627 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-home" pod "d16f785a-baaa-466b-9965-51cf8a8a0bfe" (UID: "d16f785a-baaa-466b-9965-51cf8a8a0bfe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:14.952740 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.952670 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-tmp-dir" pod "aa094251-9344-47b6-bbcf-79823b693371" (UID: "aa094251-9344-47b6-bbcf-79823b693371"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:14.952881 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.952787 5755 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-home\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:14.952881 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.952838 5755 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-tmp-dir\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:14.952881 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.952854 5755 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-tmp-dir\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:14.952881 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.952853 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-home" pod "440b1853-8437-4777-9b63-afddbbb3f89a" (UID: "440b1853-8437-4777-9b63-afddbbb3f89a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:14.953315 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.953042 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-model-cache" pod "d16f785a-baaa-466b-9965-51cf8a8a0bfe" (UID: "d16f785a-baaa-466b-9965-51cf8a8a0bfe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:14.954898 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.954699 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-dshm" pod "d16f785a-baaa-466b-9965-51cf8a8a0bfe" (UID: "d16f785a-baaa-466b-9965-51cf8a8a0bfe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:14.954898 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.954708 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16f785a-baaa-466b-9965-51cf8a8a0bfe-tls-certs" pod "d16f785a-baaa-466b-9965-51cf8a8a0bfe" (UID: "d16f785a-baaa-466b-9965-51cf8a8a0bfe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:14.954898 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.954868 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440b1853-8437-4777-9b63-afddbbb3f89a-kube-api-access-t82bw" pod "440b1853-8437-4777-9b63-afddbbb3f89a" (UID: "440b1853-8437-4777-9b63-afddbbb3f89a"). InnerVolumeSpecName "kube-api-access-t82bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:14.955114 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.954921 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16f785a-baaa-466b-9965-51cf8a8a0bfe-kube-api-access-bqshs" pod "d16f785a-baaa-466b-9965-51cf8a8a0bfe" (UID: "d16f785a-baaa-466b-9965-51cf8a8a0bfe"). InnerVolumeSpecName "kube-api-access-bqshs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:14.955114 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.955033 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa094251-9344-47b6-bbcf-79823b693371-tls-certs" pod "aa094251-9344-47b6-bbcf-79823b693371" (UID: "aa094251-9344-47b6-bbcf-79823b693371"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:14.955114 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.955086 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa094251-9344-47b6-bbcf-79823b693371-kube-api-access-dxkx5" pod "aa094251-9344-47b6-bbcf-79823b693371" (UID: "aa094251-9344-47b6-bbcf-79823b693371"). InnerVolumeSpecName "kube-api-access-dxkx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:14.955269 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.955240 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-dshm" pod "aa094251-9344-47b6-bbcf-79823b693371" (UID: "aa094251-9344-47b6-bbcf-79823b693371"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:14.955311 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.955272 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440b1853-8437-4777-9b63-afddbbb3f89a-tls-certs" pod "440b1853-8437-4777-9b63-afddbbb3f89a" (UID: "440b1853-8437-4777-9b63-afddbbb3f89a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:14.956122 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:14.956102 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-dshm" pod "440b1853-8437-4777-9b63-afddbbb3f89a" (UID: "440b1853-8437-4777-9b63-afddbbb3f89a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:15.054091 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.054060 5755 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-model-cache\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:15.054091 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.054087 5755 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa094251-9344-47b6-bbcf-79823b693371-tls-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:15.054091 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.054096 5755 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-dshm\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:15.054318 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.054106 5755 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/440b1853-8437-4777-9b63-afddbbb3f89a-tls-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:15.054318 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.054114 5755 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d16f785a-baaa-466b-9965-51cf8a8a0bfe-tls-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:15.054318 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.054123 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t82bw\" (UniqueName: \"kubernetes.io/projected/440b1853-8437-4777-9b63-afddbbb3f89a-kube-api-access-t82bw\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:15.054318 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.054134 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxkx5\" (UniqueName: \"kubernetes.io/projected/aa094251-9344-47b6-bbcf-79823b693371-kube-api-access-dxkx5\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:15.054318 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.054143 5755 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-model-cache\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:15.054318 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.054152 5755 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-dshm\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:15.054318 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.054160 5755 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-dshm\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:15.054318 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.054167 5755 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-model-cache\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:15.054318 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.054174 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqshs\" (UniqueName: \"kubernetes.io/projected/d16f785a-baaa-466b-9965-51cf8a8a0bfe-kube-api-access-bqshs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:15.054318 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.054182 5755 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa094251-9344-47b6-bbcf-79823b693371-tmp-dir\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:15.054318 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.054190 5755 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d16f785a-baaa-466b-9965-51cf8a8a0bfe-home\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:15.054318 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.054197 5755 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/440b1853-8437-4777-9b63-afddbbb3f89a-home\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:15.205079 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.205000 5755 scope.go:122] "RemoveContainer" containerID="7a637f00577070fe58599e673d6e076631ee96574e8322e1f39726368041ecdb" Oct 30 21:00:15.205079 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.205064 5755 scope.go:122] "RemoveContainer" containerID="ac3425be0aefe16d0517b7ad7e3e04b98bcbd7ccf8f3bca86209c9129749e66c" Oct 30 21:00:15.380444 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.380416 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445"] Oct 30 21:00:15.387164 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.387138 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/premium-w6-ee3fad-kserve-878b87998-l6445"] Oct 30 21:00:15.501165 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.501140 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9"] Oct 30 21:00:15.507396 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.507366 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/distinct-2-w6-ee3fad-kserve-6c74b958fd-pv5d9"] Oct 30 21:00:15.618197 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.618166 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f"] Oct 30 21:00:15.628587 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.628562 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ee3fad/simulator-w6-ee3fad-kserve-77c46d8787-mz55f"] Oct 30 21:00:15.913692 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.913611 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" event={"ID":"41470888-65ed-44de-9bcc-210cf168dece","Type":"ContainerStarted","Data":"dd2537f0e6bdea745673120b19cd06910a448fff0b76a803c11fc3d9ad255fb1"} Oct 30 21:00:15.915578 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:15.915554 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" event={"ID":"af12be6a-d0d3-47f5-bce7-65439f599fe6","Type":"ContainerStarted","Data":"0df2eeaf98abd4a2c992a785a60eddcece8eb0f19cf759f0a2a5ba24a0e7f128"} Oct 30 21:00:16.032814 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:16.032767 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" podStartSLOduration=16.032752727 podStartE2EDuration="16.032752727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:16.031377655 +0000 UTC m=+1978.957934582" watchObservedRunningTime="2026-10-30 21:00:16.032752727 +0000 UTC m=+1978.959309672" Oct 30 21:00:16.927052 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:16.927022 5755 generic.go:356] "Generic (PLEG): container finished" podID="af12be6a-d0d3-47f5-bce7-65439f599fe6" containerID="0df2eeaf98abd4a2c992a785a60eddcece8eb0f19cf759f0a2a5ba24a0e7f128" exitCode=60 Oct 30 21:00:16.927521 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:16.927096 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" event={"ID":"af12be6a-d0d3-47f5-bce7-65439f599fe6","Type":"ContainerDied","Data":"0df2eeaf98abd4a2c992a785a60eddcece8eb0f19cf759f0a2a5ba24a0e7f128"} Oct 30 21:00:16.927521 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:16.927158 5755 scope.go:122] "RemoveContainer" containerID="ac3425be0aefe16d0517b7ad7e3e04b98bcbd7ccf8f3bca86209c9129749e66c" Oct 30 21:00:16.927634 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:16.927520 5755 scope.go:122] "RemoveContainer" containerID="0df2eeaf98abd4a2c992a785a60eddcece8eb0f19cf759f0a2a5ba24a0e7f128" Oct 30 21:00:16.927776 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:16.927754 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv_odh-ai-gateway-infra(af12be6a-d0d3-47f5-bce7-65439f599fe6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" podUID="af12be6a-d0d3-47f5-bce7-65439f599fe6" Oct 30 21:00:16.929908 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:16.929882 5755 generic.go:356] "Generic (PLEG): container finished" podID="41470888-65ed-44de-9bcc-210cf168dece" containerID="dd2537f0e6bdea745673120b19cd06910a448fff0b76a803c11fc3d9ad255fb1" exitCode=60 Oct 30 21:00:16.930005 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:16.929929 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" event={"ID":"41470888-65ed-44de-9bcc-210cf168dece","Type":"ContainerDied","Data":"dd2537f0e6bdea745673120b19cd06910a448fff0b76a803c11fc3d9ad255fb1"} Oct 30 21:00:16.930301 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:16.930284 5755 scope.go:122] "RemoveContainer" containerID="dd2537f0e6bdea745673120b19cd06910a448fff0b76a803c11fc3d9ad255fb1" Oct 30 21:00:16.930622 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:16.930599 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj_odh-ai-gateway-infra(41470888-65ed-44de-9bcc-210cf168dece)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" podUID="41470888-65ed-44de-9bcc-210cf168dece" Oct 30 21:00:16.940257 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:16.940240 5755 scope.go:122] "RemoveContainer" containerID="7a637f00577070fe58599e673d6e076631ee96574e8322e1f39726368041ecdb" Oct 30 21:00:17.042296 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:17.042253 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj"] Oct 30 21:00:17.164423 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:17.164373 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv"] Oct 30 21:00:17.211705 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:17.211638 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" path="/var/lib/kubelet/pods/440b1853-8437-4777-9b63-afddbbb3f89a/volumes" Oct 30 21:00:17.212019 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:17.212006 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa094251-9344-47b6-bbcf-79823b693371" path="/var/lib/kubelet/pods/aa094251-9344-47b6-bbcf-79823b693371/volumes" Oct 30 21:00:17.212364 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:17.212352 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" path="/var/lib/kubelet/pods/d16f785a-baaa-466b-9965-51cf8a8a0bfe/volumes" Oct 30 21:00:18.055728 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.055311 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp"] Oct 30 21:00:18.056055 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.055734 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc"] Oct 30 21:00:18.079962 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.079938 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" Oct 30 21:00:18.103296 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.103269 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad6a291-s6prp"] Oct 30 21:00:18.106513 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.106494 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" Oct 30 21:00:18.109412 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.108613 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f87cb69-lw9pc"] Oct 30 21:00:18.116925 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.116904 5755 scope.go:122] "RemoveContainer" containerID="8aa1aac9a46dafccdd497867baa3daaa15da11c31a6a0b6dbea99507b937d9a6" Oct 30 21:00:18.126094 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.126075 5755 scope.go:122] "RemoveContainer" containerID="54806a3f43fe07f38bb943fa7fa85a98d779faa2e126af08af77f0c00ffc5c65" Oct 30 21:00:18.134537 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.134521 5755 scope.go:122] "RemoveContainer" containerID="d0d218d7870b7fa938e4a1cb95568c4452b400f577f8e53ce85b3268543a7c09" Oct 30 21:00:18.188706 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.188678 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af12be6a-d0d3-47f5-bce7-65439f599fe6-kube-api-access-7lfnk\" (UniqueName: \"kubernetes.io/projected/af12be6a-d0d3-47f5-bce7-65439f599fe6-kube-api-access-7lfnk\") pod \"af12be6a-d0d3-47f5-bce7-65439f599fe6\" (UID: \"af12be6a-d0d3-47f5-bce7-65439f599fe6\") " Oct 30 21:00:18.188787 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.188722 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41470888-65ed-44de-9bcc-210cf168dece-kube-api-access-r2cpt\" (UniqueName: \"kubernetes.io/projected/41470888-65ed-44de-9bcc-210cf168dece-kube-api-access-r2cpt\") pod \"41470888-65ed-44de-9bcc-210cf168dece\" (UID: \"41470888-65ed-44de-9bcc-210cf168dece\") " Oct 30 21:00:18.188787 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.188757 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41470888-65ed-44de-9bcc-210cf168dece-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41470888-65ed-44de-9bcc-210cf168dece-maas-api-service-ca\") pod \"41470888-65ed-44de-9bcc-210cf168dece\" (UID: \"41470888-65ed-44de-9bcc-210cf168dece\") " Oct 30 21:00:18.188896 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.188795 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af12be6a-d0d3-47f5-bce7-65439f599fe6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af12be6a-d0d3-47f5-bce7-65439f599fe6-maas-api-service-ca\") pod \"af12be6a-d0d3-47f5-bce7-65439f599fe6\" (UID: \"af12be6a-d0d3-47f5-bce7-65439f599fe6\") " Oct 30 21:00:18.189187 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.189155 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41470888-65ed-44de-9bcc-210cf168dece-maas-api-service-ca" pod "41470888-65ed-44de-9bcc-210cf168dece" (UID: "41470888-65ed-44de-9bcc-210cf168dece"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:18.189267 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.189190 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af12be6a-d0d3-47f5-bce7-65439f599fe6-maas-api-service-ca" pod "af12be6a-d0d3-47f5-bce7-65439f599fe6" (UID: "af12be6a-d0d3-47f5-bce7-65439f599fe6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:18.189305 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.189278 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41470888-65ed-44de-9bcc-210cf168dece-maas-api-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:18.189305 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.189295 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af12be6a-d0d3-47f5-bce7-65439f599fe6-maas-api-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:18.191007 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.190957 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41470888-65ed-44de-9bcc-210cf168dece-kube-api-access-r2cpt" pod "41470888-65ed-44de-9bcc-210cf168dece" (UID: "41470888-65ed-44de-9bcc-210cf168dece"). InnerVolumeSpecName "kube-api-access-r2cpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:18.191303 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.191271 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af12be6a-d0d3-47f5-bce7-65439f599fe6-kube-api-access-7lfnk" pod "af12be6a-d0d3-47f5-bce7-65439f599fe6" (UID: "af12be6a-d0d3-47f5-bce7-65439f599fe6"). InnerVolumeSpecName "kube-api-access-7lfnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:18.289892 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.289863 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2cpt\" (UniqueName: \"kubernetes.io/projected/41470888-65ed-44de-9bcc-210cf168dece-kube-api-access-r2cpt\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:18.289892 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.289889 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lfnk\" (UniqueName: \"kubernetes.io/projected/af12be6a-d0d3-47f5-bce7-65439f599fe6-kube-api-access-7lfnk\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:18.948599 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.948565 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" Oct 30 21:00:18.948781 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.948595 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj" event={"ID":"41470888-65ed-44de-9bcc-210cf168dece","Type":"ContainerDied","Data":"c0fcb6e2cb9c0613c69b0c3694ad49036d3bba91aa19c5de91511469d629df0b"} Oct 30 21:00:18.948781 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.948680 5755 scope.go:122] "RemoveContainer" containerID="dd2537f0e6bdea745673120b19cd06910a448fff0b76a803c11fc3d9ad255fb1" Oct 30 21:00:18.950176 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.950149 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" event={"ID":"af12be6a-d0d3-47f5-bce7-65439f599fe6","Type":"ContainerDied","Data":"9c0a81f9277f6c15e753a01bdd360d0275fb4a2b4893fad84ea537e75b7b17ed"} Oct 30 21:00:18.950300 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.950183 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv" Oct 30 21:00:18.959914 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:18.959899 5755 scope.go:122] "RemoveContainer" containerID="0df2eeaf98abd4a2c992a785a60eddcece8eb0f19cf759f0a2a5ba24a0e7f128" Oct 30 21:00:19.113837 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:19.113804 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv"] Oct 30 21:00:19.116910 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:19.116886 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889900-7qnkv"] Oct 30 21:00:19.211140 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:19.211076 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32e6b94f-518c-49b1-a079-d589db93402a" path="/var/lib/kubelet/pods/32e6b94f-518c-49b1-a079-d589db93402a/volumes" Oct 30 21:00:19.211529 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:19.211514 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="965dcdcd-dcea-4588-bec1-c4e8bd92785c" path="/var/lib/kubelet/pods/965dcdcd-dcea-4588-bec1-c4e8bd92785c/volumes" Oct 30 21:00:19.211875 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:19.211861 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af12be6a-d0d3-47f5-bce7-65439f599fe6" path="/var/lib/kubelet/pods/af12be6a-d0d3-47f5-bce7-65439f599fe6/volumes" Oct 30 21:00:19.229067 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:19.229037 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj"] Oct 30 21:00:19.236317 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:19.236295 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-19176d-29889900-qcgkj"] Oct 30 21:00:21.210560 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:21.210531 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41470888-65ed-44de-9bcc-210cf168dece" path="/var/lib/kubelet/pods/41470888-65ed-44de-9bcc-210cf168dece/volumes" Oct 30 21:00:32.366953 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:32.366878 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd"] Oct 30 21:00:32.367354 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:32.367073 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" podUID="ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" containerName="istio-proxy" containerID="cri-o://fbcf848300a612089d7ab94cc57a0c245c5e897c97455a9111626c4c78eebef6" gracePeriod=30 Oct 30 21:00:32.385485 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:32.385456 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv"] Oct 30 21:00:32.385675 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:32.385654 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerName="istio-proxy" containerID="cri-o://e5565c5246cacd2f5b8588dfb66a8e87fd57a8717835c7286e7528f719e25889" gracePeriod=30 Oct 30 21:00:37.649258 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.649191 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 21:00:37.666445 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.666416 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 21:00:37.686320 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686298 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-workload-socket\") pod \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " Oct 30 21:00:37.686436 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686334 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-credential-socket\") pod \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " Oct 30 21:00:37.686436 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686353 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-podinfo\") pod \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " Oct 30 21:00:37.686436 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686417 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-kube-api-access-cknwv\" (UniqueName: \"kubernetes.io/projected/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-kube-api-access-cknwv\") pod \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " Oct 30 21:00:37.686564 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686467 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-token\" (UniqueName: \"kubernetes.io/projected/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-token\") pod \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " Oct 30 21:00:37.686564 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686500 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-envoy\") pod \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " Oct 30 21:00:37.686564 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686529 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-credential-socket\") pod \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " Oct 30 21:00:37.686708 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686566 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-workload-certs\") pod \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " Oct 30 21:00:37.686708 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686600 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-data\") pod \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " Oct 30 21:00:37.686708 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686599 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-workload-socket" pod "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" (UID: "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:37.686708 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686637 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-workload-certs\") pod \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " Oct 30 21:00:37.686708 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686667 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istiod-ca-cert\") pod \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " Oct 30 21:00:37.686708 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686695 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-workload-socket\") pod \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " Oct 30 21:00:37.687011 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686724 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe01d860-8f2e-4a07-9e35-d25f0fa39302-kube-api-access-6gzkw\" (UniqueName: \"kubernetes.io/projected/fe01d860-8f2e-4a07-9e35-d25f0fa39302-kube-api-access-6gzkw\") pod \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " Oct 30 21:00:37.687011 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686760 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-envoy\") pod \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\" (UID: \"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a\") " Oct 30 21:00:37.687011 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686796 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-token\" (UniqueName: \"kubernetes.io/projected/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-token\") pod \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " Oct 30 21:00:37.687011 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686822 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-credential-socket" pod "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" (UID: "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:37.687011 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686831 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-workload-certs" pod "fe01d860-8f2e-4a07-9e35-d25f0fa39302" (UID: "fe01d860-8f2e-4a07-9e35-d25f0fa39302"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:37.687011 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686834 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-podinfo\") pod \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " Oct 30 21:00:37.687011 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686900 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istiod-ca-cert\") pod \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " Oct 30 21:00:37.687011 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.686929 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-data\") pod \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\" (UID: \"fe01d860-8f2e-4a07-9e35-d25f0fa39302\") " Oct 30 21:00:37.687011 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.687009 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-data" pod "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" (UID: "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:37.687672 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.687133 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-credential-socket" pod "fe01d860-8f2e-4a07-9e35-d25f0fa39302" (UID: "fe01d860-8f2e-4a07-9e35-d25f0fa39302"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:37.687672 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.687303 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-workload-socket" pod "fe01d860-8f2e-4a07-9e35-d25f0fa39302" (UID: "fe01d860-8f2e-4a07-9e35-d25f0fa39302"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:37.687788 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.687676 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-workload-certs" pod "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" (UID: "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:37.688700 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.688499 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-data" pod "fe01d860-8f2e-4a07-9e35-d25f0fa39302" (UID: "fe01d860-8f2e-4a07-9e35-d25f0fa39302"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:37.688700 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.688632 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istiod-ca-cert" pod "fe01d860-8f2e-4a07-9e35-d25f0fa39302" (UID: "fe01d860-8f2e-4a07-9e35-d25f0fa39302"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:37.688700 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.688658 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istiod-ca-cert" pod "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" (UID: "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:37.688972 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.688787 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.688972 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.688805 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.688972 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.688820 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.688972 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.688833 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.688972 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.688846 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.688972 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.688859 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.689471 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.689189 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-envoy" pod "fe01d860-8f2e-4a07-9e35-d25f0fa39302" (UID: "fe01d860-8f2e-4a07-9e35-d25f0fa39302"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:37.689755 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.689663 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-podinfo" pod "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" (UID: "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:37.689755 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.689722 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-podinfo" pod "fe01d860-8f2e-4a07-9e35-d25f0fa39302" (UID: "fe01d860-8f2e-4a07-9e35-d25f0fa39302"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:37.689895 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.689841 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-token" pod "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" (UID: "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:37.689895 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.689855 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe01d860-8f2e-4a07-9e35-d25f0fa39302-kube-api-access-6gzkw" pod "fe01d860-8f2e-4a07-9e35-d25f0fa39302" (UID: "fe01d860-8f2e-4a07-9e35-d25f0fa39302"). InnerVolumeSpecName "kube-api-access-6gzkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:37.690496 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.690470 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-kube-api-access-cknwv" pod "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" (UID: "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a"). InnerVolumeSpecName "kube-api-access-cknwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:37.690751 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.690730 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-token" pod "fe01d860-8f2e-4a07-9e35-d25f0fa39302" (UID: "fe01d860-8f2e-4a07-9e35-d25f0fa39302"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:37.690971 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.690951 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-envoy" pod "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" (UID: "ffff493d-18a3-4ca2-adc9-3bdc3b51c30a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:37.789961 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.789936 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.789961 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.789958 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cknwv\" (UniqueName: \"kubernetes.io/projected/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-kube-api-access-cknwv\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.790104 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.789969 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.790104 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.789978 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.790104 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.789987 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.790104 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.789996 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.790104 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.790005 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gzkw\" (UniqueName: \"kubernetes.io/projected/fe01d860-8f2e-4a07-9e35-d25f0fa39302-kube-api-access-6gzkw\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.790104 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.790013 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.790104 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.790022 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.790104 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.790030 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.790104 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.790039 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.790104 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:37.790048 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe01d860-8f2e-4a07-9e35-d25f0fa39302-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.069005 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.068970 5755 generic.go:356] "Generic (PLEG): container finished" podID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerID="e5565c5246cacd2f5b8588dfb66a8e87fd57a8717835c7286e7528f719e25889" exitCode=0 Oct 30 21:00:38.069221 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.069030 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" Oct 30 21:00:38.069221 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.069053 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" event={"ID":"fe01d860-8f2e-4a07-9e35-d25f0fa39302","Type":"ContainerDied","Data":"e5565c5246cacd2f5b8588dfb66a8e87fd57a8717835c7286e7528f719e25889"} Oct 30 21:00:38.069221 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.069118 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv" event={"ID":"fe01d860-8f2e-4a07-9e35-d25f0fa39302","Type":"ContainerDied","Data":"a4b6f4618bad645cc050154f66fdb7825d9d2d09b2045b95b6af2ffba450c3a7"} Oct 30 21:00:38.069221 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.069150 5755 scope.go:122] "RemoveContainer" containerID="e5565c5246cacd2f5b8588dfb66a8e87fd57a8717835c7286e7528f719e25889" Oct 30 21:00:38.070760 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.070733 5755 generic.go:356] "Generic (PLEG): container finished" podID="ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" containerID="fbcf848300a612089d7ab94cc57a0c245c5e897c97455a9111626c4c78eebef6" exitCode=0 Oct 30 21:00:38.070903 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.070806 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" Oct 30 21:00:38.070987 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.070812 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" event={"ID":"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a","Type":"ContainerDied","Data":"fbcf848300a612089d7ab94cc57a0c245c5e897c97455a9111626c4c78eebef6"} Oct 30 21:00:38.071044 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.071029 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd" event={"ID":"ffff493d-18a3-4ca2-adc9-3bdc3b51c30a","Type":"ContainerDied","Data":"221aeb4591a6acb910e4290d75e6472af4f67c690b17ea95ebd7726184061a98"} Oct 30 21:00:38.080106 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.080007 5755 scope.go:122] "RemoveContainer" containerID="e5565c5246cacd2f5b8588dfb66a8e87fd57a8717835c7286e7528f719e25889" Oct 30 21:00:38.080282 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:38.080262 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5565c5246cacd2f5b8588dfb66a8e87fd57a8717835c7286e7528f719e25889\": container with ID starting with e5565c5246cacd2f5b8588dfb66a8e87fd57a8717835c7286e7528f719e25889 not found: ID does not exist" containerID="e5565c5246cacd2f5b8588dfb66a8e87fd57a8717835c7286e7528f719e25889" Oct 30 21:00:38.080357 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.080295 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5565c5246cacd2f5b8588dfb66a8e87fd57a8717835c7286e7528f719e25889"} err="failed to get container status \"e5565c5246cacd2f5b8588dfb66a8e87fd57a8717835c7286e7528f719e25889\": rpc error: code = NotFound desc = could not find container \"e5565c5246cacd2f5b8588dfb66a8e87fd57a8717835c7286e7528f719e25889\": container with ID starting with e5565c5246cacd2f5b8588dfb66a8e87fd57a8717835c7286e7528f719e25889 not found: ID does not exist" Oct 30 21:00:38.080357 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.080316 5755 scope.go:122] "RemoveContainer" containerID="fbcf848300a612089d7ab94cc57a0c245c5e897c97455a9111626c4c78eebef6" Oct 30 21:00:38.088740 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.088722 5755 scope.go:122] "RemoveContainer" containerID="fbcf848300a612089d7ab94cc57a0c245c5e897c97455a9111626c4c78eebef6" Oct 30 21:00:38.089002 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:38.088982 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbcf848300a612089d7ab94cc57a0c245c5e897c97455a9111626c4c78eebef6\": container with ID starting with fbcf848300a612089d7ab94cc57a0c245c5e897c97455a9111626c4c78eebef6 not found: ID does not exist" containerID="fbcf848300a612089d7ab94cc57a0c245c5e897c97455a9111626c4c78eebef6" Oct 30 21:00:38.089094 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.089008 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbcf848300a612089d7ab94cc57a0c245c5e897c97455a9111626c4c78eebef6"} err="failed to get container status \"fbcf848300a612089d7ab94cc57a0c245c5e897c97455a9111626c4c78eebef6\": rpc error: code = NotFound desc = could not find container \"fbcf848300a612089d7ab94cc57a0c245c5e897c97455a9111626c4c78eebef6\": container with ID starting with fbcf848300a612089d7ab94cc57a0c245c5e897c97455a9111626c4c78eebef6 not found: ID does not exist" Oct 30 21:00:38.235876 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.235847 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv"] Oct 30 21:00:38.243087 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.243061 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-v6rbv"] Oct 30 21:00:38.352956 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.352883 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd"] Oct 30 21:00:38.359779 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:38.359754 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-mfdfd"] Oct 30 21:00:39.210732 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:39.210699 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" path="/var/lib/kubelet/pods/fe01d860-8f2e-4a07-9e35-d25f0fa39302/volumes" Oct 30 21:00:39.211118 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:39.211102 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" path="/var/lib/kubelet/pods/ffff493d-18a3-4ca2-adc9-3bdc3b51c30a/volumes" Oct 30 21:00:40.186246 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:40.186210 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh"] Oct 30 21:00:41.255599 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:41.255563 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:00:55.206322 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.206272 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" containerID="cri-o://c41d2c3effc96de25720ea854ac7db321953e37b4c9cc9daea516230d82699a1" gracePeriod=45 Oct 30 21:00:55.454932 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.454903 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 21:00:55.559209 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.559164 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-home\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-home\") pod \"0be44dfb-bf6d-48a8-b061-71899ad1587f\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " Oct 30 21:00:55.559437 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.559232 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be44dfb-bf6d-48a8-b061-71899ad1587f-kube-api-access-5mdn7\" (UniqueName: \"kubernetes.io/projected/0be44dfb-bf6d-48a8-b061-71899ad1587f-kube-api-access-5mdn7\") pod \"0be44dfb-bf6d-48a8-b061-71899ad1587f\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " Oct 30 21:00:55.559437 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.559276 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-tmp-dir\") pod \"0be44dfb-bf6d-48a8-b061-71899ad1587f\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " Oct 30 21:00:55.559437 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.559317 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0be44dfb-bf6d-48a8-b061-71899ad1587f-tls-certs\" (UniqueName: \"kubernetes.io/secret/0be44dfb-bf6d-48a8-b061-71899ad1587f-tls-certs\") pod \"0be44dfb-bf6d-48a8-b061-71899ad1587f\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " Oct 30 21:00:55.559437 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.559349 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-model-cache\") pod \"0be44dfb-bf6d-48a8-b061-71899ad1587f\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " Oct 30 21:00:55.559437 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.559423 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-dshm\") pod \"0be44dfb-bf6d-48a8-b061-71899ad1587f\" (UID: \"0be44dfb-bf6d-48a8-b061-71899ad1587f\") " Oct 30 21:00:55.559722 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.559462 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-home" pod "0be44dfb-bf6d-48a8-b061-71899ad1587f" (UID: "0be44dfb-bf6d-48a8-b061-71899ad1587f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:55.559722 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.559576 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-tmp-dir" pod "0be44dfb-bf6d-48a8-b061-71899ad1587f" (UID: "0be44dfb-bf6d-48a8-b061-71899ad1587f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:55.559722 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.559631 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-model-cache" pod "0be44dfb-bf6d-48a8-b061-71899ad1587f" (UID: "0be44dfb-bf6d-48a8-b061-71899ad1587f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:55.559856 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.559777 5755 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-home\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:55.559856 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.559793 5755 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-tmp-dir\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:55.559856 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.559806 5755 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-model-cache\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:55.561730 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.561705 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-dshm" pod "0be44dfb-bf6d-48a8-b061-71899ad1587f" (UID: "0be44dfb-bf6d-48a8-b061-71899ad1587f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:55.561837 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.561777 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0be44dfb-bf6d-48a8-b061-71899ad1587f-tls-certs" pod "0be44dfb-bf6d-48a8-b061-71899ad1587f" (UID: "0be44dfb-bf6d-48a8-b061-71899ad1587f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:55.561892 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.561870 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be44dfb-bf6d-48a8-b061-71899ad1587f-kube-api-access-5mdn7" pod "0be44dfb-bf6d-48a8-b061-71899ad1587f" (UID: "0be44dfb-bf6d-48a8-b061-71899ad1587f"). InnerVolumeSpecName "kube-api-access-5mdn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:55.660557 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.660527 5755 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0be44dfb-bf6d-48a8-b061-71899ad1587f-tls-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:55.660557 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.660556 5755 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0be44dfb-bf6d-48a8-b061-71899ad1587f-dshm\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:55.660757 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.660567 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mdn7\" (UniqueName: \"kubernetes.io/projected/0be44dfb-bf6d-48a8-b061-71899ad1587f-kube-api-access-5mdn7\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:55.733067 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733031 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj"] Oct 30 21:00:55.733447 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733426 5755 cpu_manager.go:414] "containerMap: removing container" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerName="istio-proxy" Oct 30 21:00:55.733504 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733448 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerName="istio-proxy" Oct 30 21:00:55.733504 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733473 5755 cpu_manager.go:414] "containerMap: removing container" podUID="16840665-c3f7-48bf-a2bc-695c0be23df0" containerName="revoke-keys" Oct 30 21:00:55.733504 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733478 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16840665-c3f7-48bf-a2bc-695c0be23df0" containerName="revoke-keys" Oct 30 21:00:55.733504 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733486 5755 cpu_manager.go:414] "containerMap: removing container" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" Oct 30 21:00:55.733504 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733491 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" Oct 30 21:00:55.733504 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733500 5755 cpu_manager.go:414] "containerMap: removing container" podUID="41470888-65ed-44de-9bcc-210cf168dece" containerName="cleanup" Oct 30 21:00:55.733504 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733504 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41470888-65ed-44de-9bcc-210cf168dece" containerName="cleanup" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733515 5755 cpu_manager.go:414] "containerMap: removing container" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733519 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733526 5755 cpu_manager.go:414] "containerMap: removing container" podUID="ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" containerName="istio-proxy" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733531 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" containerName="istio-proxy" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733543 5755 cpu_manager.go:414] "containerMap: removing container" podUID="41470888-65ed-44de-9bcc-210cf168dece" containerName="cleanup" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733548 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41470888-65ed-44de-9bcc-210cf168dece" containerName="cleanup" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733554 5755 cpu_manager.go:414] "containerMap: removing container" podUID="41470888-65ed-44de-9bcc-210cf168dece" containerName="cleanup" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733558 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41470888-65ed-44de-9bcc-210cf168dece" containerName="cleanup" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733564 5755 cpu_manager.go:414] "containerMap: removing container" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733568 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733574 5755 cpu_manager.go:414] "containerMap: removing container" podUID="af12be6a-d0d3-47f5-bce7-65439f599fe6" containerName="cleanup" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733578 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af12be6a-d0d3-47f5-bce7-65439f599fe6" containerName="cleanup" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733586 5755 cpu_manager.go:414] "containerMap: removing container" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733590 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733596 5755 cpu_manager.go:414] "containerMap: removing container" podUID="af12be6a-d0d3-47f5-bce7-65439f599fe6" containerName="cleanup" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733600 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af12be6a-d0d3-47f5-bce7-65439f599fe6" containerName="cleanup" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733606 5755 cpu_manager.go:414] "containerMap: removing container" podUID="af12be6a-d0d3-47f5-bce7-65439f599fe6" containerName="cleanup" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733610 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af12be6a-d0d3-47f5-bce7-65439f599fe6" containerName="cleanup" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733661 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerName="main" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733669 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffff493d-18a3-4ca2-adc9-3bdc3b51c30a" containerName="istio-proxy" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733676 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="41470888-65ed-44de-9bcc-210cf168dece" containerName="cleanup" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733682 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="41470888-65ed-44de-9bcc-210cf168dece" containerName="cleanup" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733688 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16f785a-baaa-466b-9965-51cf8a8a0bfe" containerName="main" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733693 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="af12be6a-d0d3-47f5-bce7-65439f599fe6" containerName="cleanup" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733698 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa094251-9344-47b6-bbcf-79823b693371" containerName="main" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733703 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="af12be6a-d0d3-47f5-bce7-65439f599fe6" containerName="cleanup" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733708 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="af12be6a-d0d3-47f5-bce7-65439f599fe6" containerName="cleanup" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733713 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe01d860-8f2e-4a07-9e35-d25f0fa39302" containerName="istio-proxy" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733721 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="16840665-c3f7-48bf-a2bc-695c0be23df0" containerName="revoke-keys" Oct 30 21:00:55.733717 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.733727 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="440b1853-8437-4777-9b63-afddbbb3f89a" containerName="main" Oct 30 21:00:55.736989 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.736962 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" Oct 30 21:00:55.794341 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.794305 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj"] Oct 30 21:00:55.834533 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.834455 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:00:55.834533 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.834501 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 21:00:55.862829 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.862791 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3df662a7-hrlgj\" (UID: \"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" Oct 30 21:00:55.862996 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.862861 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3df662a7-hrlgj\" (UID: \"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" Oct 30 21:00:55.862996 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.862991 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl6jb\" (UniqueName: \"kubernetes.io/projected/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-kube-api-access-tl6jb\") pod \"maas-api-revoke-sub-3df662a7-hrlgj\" (UID: \"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" Oct 30 21:00:55.963559 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.963524 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3df662a7-hrlgj\" (UID: \"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" Oct 30 21:00:55.963711 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.963583 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl6jb\" (UniqueName: \"kubernetes.io/projected/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-kube-api-access-tl6jb\") pod \"maas-api-revoke-sub-3df662a7-hrlgj\" (UID: \"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" Oct 30 21:00:55.963711 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.963618 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3df662a7-hrlgj\" (UID: \"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" Oct 30 21:00:55.964211 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.964188 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3df662a7-hrlgj\" (UID: \"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" Oct 30 21:00:55.965970 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:55.965952 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3df662a7-hrlgj\" (UID: \"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" Oct 30 21:00:56.021850 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:56.021815 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl6jb\" (UniqueName: \"kubernetes.io/projected/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-kube-api-access-tl6jb\") pod \"maas-api-revoke-sub-3df662a7-hrlgj\" (UID: \"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" Oct 30 21:00:56.052806 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:56.052782 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" Oct 30 21:00:56.192736 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:56.192703 5755 generic.go:356] "Generic (PLEG): container finished" podID="0be44dfb-bf6d-48a8-b061-71899ad1587f" containerID="c41d2c3effc96de25720ea854ac7db321953e37b4c9cc9daea516230d82699a1" exitCode=0 Oct 30 21:00:56.192923 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:56.192752 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" Oct 30 21:00:56.192923 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:56.192799 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" event={"ID":"0be44dfb-bf6d-48a8-b061-71899ad1587f","Type":"ContainerDied","Data":"c41d2c3effc96de25720ea854ac7db321953e37b4c9cc9daea516230d82699a1"} Oct 30 21:00:56.192923 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:56.192834 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh" event={"ID":"0be44dfb-bf6d-48a8-b061-71899ad1587f","Type":"ContainerDied","Data":"fce01604081d808af25751fb3c3a4e1ce5cf25ac74f87ee3fd4f6051415fcc22"} Oct 30 21:00:56.192923 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:56.192851 5755 scope.go:122] "RemoveContainer" containerID="c41d2c3effc96de25720ea854ac7db321953e37b4c9cc9daea516230d82699a1" Oct 30 21:00:56.202468 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:56.202453 5755 scope.go:122] "RemoveContainer" containerID="c41d2c3effc96de25720ea854ac7db321953e37b4c9cc9daea516230d82699a1" Oct 30 21:00:56.202730 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:00:56.202713 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c41d2c3effc96de25720ea854ac7db321953e37b4c9cc9daea516230d82699a1\": container with ID starting with c41d2c3effc96de25720ea854ac7db321953e37b4c9cc9daea516230d82699a1 not found: ID does not exist" containerID="c41d2c3effc96de25720ea854ac7db321953e37b4c9cc9daea516230d82699a1" Oct 30 21:00:56.202806 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:56.202737 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41d2c3effc96de25720ea854ac7db321953e37b4c9cc9daea516230d82699a1"} err="failed to get container status \"c41d2c3effc96de25720ea854ac7db321953e37b4c9cc9daea516230d82699a1\": rpc error: code = NotFound desc = could not find container \"c41d2c3effc96de25720ea854ac7db321953e37b4c9cc9daea516230d82699a1\": container with ID starting with c41d2c3effc96de25720ea854ac7db321953e37b4c9cc9daea516230d82699a1 not found: ID does not exist" Oct 30 21:00:56.360099 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:56.360002 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh"] Oct 30 21:00:56.362754 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:56.362728 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-395022/distinct-2-w4-395022-kserve-786f767c59-pvskh"] Oct 30 21:00:56.471106 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:56.471085 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj"] Oct 30 21:00:56.472955 ip-10-0-139-195 kubenswrapper[5755]: W1030 21:00:56.472928 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a6c2ff_fa1a_4b2c_9f80_d2a0757767e5.slice/crio-cffac152996e3f1ab0fec783d66d6457bb8bc0856746144aa9fdf1b8700d710f WatchSource:0}: Error finding container cffac152996e3f1ab0fec783d66d6457bb8bc0856746144aa9fdf1b8700d710f: Status 404 returned error can't find the container with id cffac152996e3f1ab0fec783d66d6457bb8bc0856746144aa9fdf1b8700d710f Oct 30 21:00:57.201253 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:57.201213 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" event={"ID":"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5","Type":"ContainerStarted","Data":"5d5df5a1f706dec22568d936b612f1d4bdd7b3e05a38ae5bd325fe10e7e0c6f7"} Oct 30 21:00:57.201469 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:57.201284 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" event={"ID":"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5","Type":"ContainerStarted","Data":"cffac152996e3f1ab0fec783d66d6457bb8bc0856746144aa9fdf1b8700d710f"} Oct 30 21:00:57.211064 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:57.211040 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0be44dfb-bf6d-48a8-b061-71899ad1587f" path="/var/lib/kubelet/pods/0be44dfb-bf6d-48a8-b061-71899ad1587f/volumes" Oct 30 21:00:57.318036 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:57.317968 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" podStartSLOduration=2.317949084 podStartE2EDuration="2.317949084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:57.315121714 +0000 UTC m=+2020.241678631" watchObservedRunningTime="2026-10-30 21:00:57.317949084 +0000 UTC m=+2020.244506001" Oct 30 21:00:58.209701 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:58.209661 5755 generic.go:356] "Generic (PLEG): container finished" podID="73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5" containerID="5d5df5a1f706dec22568d936b612f1d4bdd7b3e05a38ae5bd325fe10e7e0c6f7" exitCode=0 Oct 30 21:00:58.210135 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:58.209728 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" event={"ID":"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5","Type":"ContainerDied","Data":"5d5df5a1f706dec22568d936b612f1d4bdd7b3e05a38ae5bd325fe10e7e0c6f7"} Oct 30 21:00:59.345770 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:59.345743 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" Oct 30 21:00:59.498304 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:59.498273 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-kube-api-access-tl6jb\" (UniqueName: \"kubernetes.io/projected/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-kube-api-access-tl6jb\") pod \"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5\" (UID: \"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5\") " Oct 30 21:00:59.498525 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:59.498309 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-maas-api-openshift-ca\") pod \"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5\" (UID: \"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5\") " Oct 30 21:00:59.498525 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:59.498335 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-maas-api-xks-ca\") pod \"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5\" (UID: \"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5\") " Oct 30 21:00:59.498722 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:59.498698 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-maas-api-openshift-ca" pod "73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5" (UID: "73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:59.500589 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:59.500567 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-maas-api-xks-ca" pod "73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5" (UID: "73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:59.500681 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:59.500654 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-kube-api-access-tl6jb" pod "73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5" (UID: "73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5"). InnerVolumeSpecName "kube-api-access-tl6jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:59.599633 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:59.599596 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl6jb\" (UniqueName: \"kubernetes.io/projected/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-kube-api-access-tl6jb\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:59.599633 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:59.599627 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:59.599633 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:00:59.599641 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:00.221955 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:00.221919 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" Oct 30 21:01:00.222135 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:00.221953 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj" event={"ID":"73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5","Type":"ContainerDied","Data":"cffac152996e3f1ab0fec783d66d6457bb8bc0856746144aa9fdf1b8700d710f"} Oct 30 21:01:00.222135 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:00.221996 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cffac152996e3f1ab0fec783d66d6457bb8bc0856746144aa9fdf1b8700d710f" Oct 30 21:01:11.262033 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:01:11.261970 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:01:18.235801 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:18.235767 5755 scope.go:122] "RemoveContainer" containerID="d3611f6c63d4df8442a2616d16f4d9868dc036d2e42a9ec08e2cfcce190916c8" Oct 30 21:01:18.246667 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:18.246643 5755 scope.go:122] "RemoveContainer" containerID="dcadbc61171cb2c64151b5a8af952fdc783b5795cc3f21d7bdc520a6b3f3f8a0" Oct 30 21:01:18.255612 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:18.255592 5755 scope.go:122] "RemoveContainer" containerID="20b3dce11b28b2c6c34d0a825f7c494a4e69a2d2fac49ab4ee5efad791411ebb" Oct 30 21:01:18.263739 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:18.263716 5755 scope.go:122] "RemoveContainer" containerID="cae441a5ef6bee1b07345280a943e9118802ec614dc050bbe5c287c3500c926f" Oct 30 21:01:21.843095 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:21.843062 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4"] Oct 30 21:01:21.843537 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:21.843291 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" podUID="cd8a8354-e543-4fe8-8385-de8b3ea8b417" containerName="istio-proxy" containerID="cri-o://00772390dfe7987a298742b10a6f6bb0c36d13bc7946f895cab6a833d7d8a683" gracePeriod=30 Oct 30 21:01:21.856834 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:21.856804 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56"] Oct 30 21:01:21.857015 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:21.856995 5755 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" podUID="ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" containerName="istio-proxy" containerID="cri-o://ce4ff84590afad8ccf5ac2337fe4eaa809691d4f7b0a2855576b07b558a3ead6" gracePeriod=30 Oct 30 21:01:25.311962 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:25.311926 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:01:25.312361 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:25.311995 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" podUID="ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:01:25.525369 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:25.525331 5755 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:01:25.525575 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:25.525412 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" podUID="cd8a8354-e543-4fe8-8385-de8b3ea8b417" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:01:27.125224 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.125199 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 21:01:27.128008 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.127989 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 21:01:27.158000 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.157974 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-credential-socket\") pod \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " Oct 30 21:01:27.158145 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158007 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-podinfo\") pod \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " Oct 30 21:01:27.158145 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158026 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istiod-ca-cert\") pod \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " Oct 30 21:01:27.158145 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158046 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istiod-ca-cert\") pod \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " Oct 30 21:01:27.158145 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158067 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-workload-socket\") pod \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " Oct 30 21:01:27.158145 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158093 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-workload-certs\") pod \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " Oct 30 21:01:27.158363 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158147 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-workload-certs\") pod \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " Oct 30 21:01:27.158363 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158179 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-workload-socket\") pod \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " Oct 30 21:01:27.158363 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158207 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-envoy\") pod \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " Oct 30 21:01:27.158363 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158243 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-token\" (UniqueName: \"kubernetes.io/projected/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-token\") pod \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " Oct 30 21:01:27.158363 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158267 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-podinfo\") pod \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " Oct 30 21:01:27.158363 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158314 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-credential-socket" pod "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" (UID: "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:27.158363 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158320 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-data\") pod \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " Oct 30 21:01:27.158753 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158435 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-token\" (UniqueName: \"kubernetes.io/projected/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-token\") pod \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " Oct 30 21:01:27.158753 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158480 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd8a8354-e543-4fe8-8385-de8b3ea8b417-kube-api-access-jmfrh\" (UniqueName: \"kubernetes.io/projected/cd8a8354-e543-4fe8-8385-de8b3ea8b417-kube-api-access-jmfrh\") pod \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " Oct 30 21:01:27.158753 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158524 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-data\") pod \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " Oct 30 21:01:27.158753 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158560 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-envoy\") pod \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " Oct 30 21:01:27.158753 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158590 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-credential-socket\") pod \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\" (UID: \"cd8a8354-e543-4fe8-8385-de8b3ea8b417\") " Oct 30 21:01:27.158753 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158639 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-kube-api-access-vd7gf\" (UniqueName: \"kubernetes.io/projected/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-kube-api-access-vd7gf\") pod \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\" (UID: \"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7\") " Oct 30 21:01:27.159079 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158980 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.159079 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158433 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-workload-socket" pod "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" (UID: "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:27.159189 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158440 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istiod-ca-cert" pod "cd8a8354-e543-4fe8-8385-de8b3ea8b417" (UID: "cd8a8354-e543-4fe8-8385-de8b3ea8b417"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:27.159189 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158569 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-workload-certs" pod "cd8a8354-e543-4fe8-8385-de8b3ea8b417" (UID: "cd8a8354-e543-4fe8-8385-de8b3ea8b417"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:27.159189 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158622 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-workload-certs" pod "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" (UID: "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:27.159189 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158696 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istiod-ca-cert" pod "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" (UID: "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:27.159189 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.158795 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-workload-socket" pod "cd8a8354-e543-4fe8-8385-de8b3ea8b417" (UID: "cd8a8354-e543-4fe8-8385-de8b3ea8b417"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:27.159189 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.159050 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-data" pod "cd8a8354-e543-4fe8-8385-de8b3ea8b417" (UID: "cd8a8354-e543-4fe8-8385-de8b3ea8b417"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:27.160716 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.160667 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-credential-socket" pod "cd8a8354-e543-4fe8-8385-de8b3ea8b417" (UID: "cd8a8354-e543-4fe8-8385-de8b3ea8b417"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:27.160716 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.160708 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-data" pod "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" (UID: "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:27.161078 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.161053 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-token" pod "cd8a8354-e543-4fe8-8385-de8b3ea8b417" (UID: "cd8a8354-e543-4fe8-8385-de8b3ea8b417"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:27.161321 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.161291 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-podinfo" pod "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" (UID: "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:01:27.161321 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.161303 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8a8354-e543-4fe8-8385-de8b3ea8b417-kube-api-access-jmfrh" pod "cd8a8354-e543-4fe8-8385-de8b3ea8b417" (UID: "cd8a8354-e543-4fe8-8385-de8b3ea8b417"). InnerVolumeSpecName "kube-api-access-jmfrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:27.161459 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.161319 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-podinfo" pod "cd8a8354-e543-4fe8-8385-de8b3ea8b417" (UID: "cd8a8354-e543-4fe8-8385-de8b3ea8b417"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:01:27.161584 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.161570 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-token" pod "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" (UID: "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:27.161663 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.161649 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-kube-api-access-vd7gf" pod "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" (UID: "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7"). InnerVolumeSpecName "kube-api-access-vd7gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:27.162129 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.162107 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-envoy" pod "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" (UID: "ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:27.162244 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.162229 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-envoy" pod "cd8a8354-e543-4fe8-8385-de8b3ea8b417" (UID: "cd8a8354-e543-4fe8-8385-de8b3ea8b417"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:27.259882 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.259852 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.259882 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.259880 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.259882 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.259891 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.260120 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.259899 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.260120 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.259907 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.260120 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.259914 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.260120 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.259922 5755 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-token\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.260120 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.259930 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmfrh\" (UniqueName: \"kubernetes.io/projected/cd8a8354-e543-4fe8-8385-de8b3ea8b417-kube-api-access-jmfrh\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.260120 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.259940 5755 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-data\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.260120 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.259947 5755 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istio-envoy\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.260120 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.259955 5755 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8a8354-e543-4fe8-8385-de8b3ea8b417-credential-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.260120 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.259962 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd7gf\" (UniqueName: \"kubernetes.io/projected/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-kube-api-access-vd7gf\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.260120 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.259970 5755 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istio-podinfo\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.260120 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.259977 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8a8354-e543-4fe8-8385-de8b3ea8b417-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.260120 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.259985 5755 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-istiod-ca-cert\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.260120 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.259993 5755 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-workload-socket\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.260120 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.260002 5755 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7-workload-certs\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:27.384781 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.384686 5755 generic.go:356] "Generic (PLEG): container finished" podID="cd8a8354-e543-4fe8-8385-de8b3ea8b417" containerID="00772390dfe7987a298742b10a6f6bb0c36d13bc7946f895cab6a833d7d8a683" exitCode=0 Oct 30 21:01:27.384781 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.384743 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" Oct 30 21:01:27.384991 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.384781 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" event={"ID":"cd8a8354-e543-4fe8-8385-de8b3ea8b417","Type":"ContainerDied","Data":"00772390dfe7987a298742b10a6f6bb0c36d13bc7946f895cab6a833d7d8a683"} Oct 30 21:01:27.384991 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.384837 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4" event={"ID":"cd8a8354-e543-4fe8-8385-de8b3ea8b417","Type":"ContainerDied","Data":"a73d3af427cf5fde4585046e987333b8e5cadaa109139d8bf6735b9ee71768e9"} Oct 30 21:01:27.384991 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.384861 5755 scope.go:122] "RemoveContainer" containerID="00772390dfe7987a298742b10a6f6bb0c36d13bc7946f895cab6a833d7d8a683" Oct 30 21:01:27.386474 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.386451 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" Oct 30 21:01:27.386474 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.386469 5755 generic.go:356] "Generic (PLEG): container finished" podID="ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" containerID="ce4ff84590afad8ccf5ac2337fe4eaa809691d4f7b0a2855576b07b558a3ead6" exitCode=0 Oct 30 21:01:27.386684 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.386500 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" event={"ID":"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7","Type":"ContainerDied","Data":"ce4ff84590afad8ccf5ac2337fe4eaa809691d4f7b0a2855576b07b558a3ead6"} Oct 30 21:01:27.386684 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.386541 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56" event={"ID":"ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7","Type":"ContainerDied","Data":"b1f5db68c6c3e47f49fffdf81c7e6735a026ab5c7de97bc3518cc32240a19a51"} Oct 30 21:01:27.394445 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.394422 5755 scope.go:122] "RemoveContainer" containerID="00772390dfe7987a298742b10a6f6bb0c36d13bc7946f895cab6a833d7d8a683" Oct 30 21:01:27.394761 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:01:27.394740 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00772390dfe7987a298742b10a6f6bb0c36d13bc7946f895cab6a833d7d8a683\": container with ID starting with 00772390dfe7987a298742b10a6f6bb0c36d13bc7946f895cab6a833d7d8a683 not found: ID does not exist" containerID="00772390dfe7987a298742b10a6f6bb0c36d13bc7946f895cab6a833d7d8a683" Oct 30 21:01:27.394818 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.394774 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00772390dfe7987a298742b10a6f6bb0c36d13bc7946f895cab6a833d7d8a683"} err="failed to get container status \"00772390dfe7987a298742b10a6f6bb0c36d13bc7946f895cab6a833d7d8a683\": rpc error: code = NotFound desc = could not find container \"00772390dfe7987a298742b10a6f6bb0c36d13bc7946f895cab6a833d7d8a683\": container with ID starting with 00772390dfe7987a298742b10a6f6bb0c36d13bc7946f895cab6a833d7d8a683 not found: ID does not exist" Oct 30 21:01:27.394818 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.394798 5755 scope.go:122] "RemoveContainer" containerID="ce4ff84590afad8ccf5ac2337fe4eaa809691d4f7b0a2855576b07b558a3ead6" Oct 30 21:01:27.403330 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.403312 5755 scope.go:122] "RemoveContainer" containerID="ce4ff84590afad8ccf5ac2337fe4eaa809691d4f7b0a2855576b07b558a3ead6" Oct 30 21:01:27.403621 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:01:27.403602 5755 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4ff84590afad8ccf5ac2337fe4eaa809691d4f7b0a2855576b07b558a3ead6\": container with ID starting with ce4ff84590afad8ccf5ac2337fe4eaa809691d4f7b0a2855576b07b558a3ead6 not found: ID does not exist" containerID="ce4ff84590afad8ccf5ac2337fe4eaa809691d4f7b0a2855576b07b558a3ead6" Oct 30 21:01:27.403699 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.403639 5755 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4ff84590afad8ccf5ac2337fe4eaa809691d4f7b0a2855576b07b558a3ead6"} err="failed to get container status \"ce4ff84590afad8ccf5ac2337fe4eaa809691d4f7b0a2855576b07b558a3ead6\": rpc error: code = NotFound desc = could not find container \"ce4ff84590afad8ccf5ac2337fe4eaa809691d4f7b0a2855576b07b558a3ead6\": container with ID starting with ce4ff84590afad8ccf5ac2337fe4eaa809691d4f7b0a2855576b07b558a3ead6 not found: ID does not exist" Oct 30 21:01:27.588365 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.588331 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4"] Oct 30 21:01:27.596963 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.596931 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-m22z4"] Oct 30 21:01:27.707776 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.707706 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56"] Oct 30 21:01:27.716629 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:27.716601 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-qbz56"] Oct 30 21:01:29.213423 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:29.213363 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd8a8354-e543-4fe8-8385-de8b3ea8b417" path="/var/lib/kubelet/pods/cd8a8354-e543-4fe8-8385-de8b3ea8b417/volumes" Oct 30 21:01:29.214056 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:29.214032 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" path="/var/lib/kubelet/pods/ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7/volumes" Oct 30 21:01:41.196973 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.196928 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s"] Oct 30 21:01:41.197510 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.197472 5755 cpu_manager.go:414] "containerMap: removing container" podUID="cd8a8354-e543-4fe8-8385-de8b3ea8b417" containerName="istio-proxy" Oct 30 21:01:41.197510 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.197491 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd8a8354-e543-4fe8-8385-de8b3ea8b417" containerName="istio-proxy" Oct 30 21:01:41.197649 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.197533 5755 cpu_manager.go:414] "containerMap: removing container" podUID="ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" containerName="istio-proxy" Oct 30 21:01:41.197649 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.197541 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" containerName="istio-proxy" Oct 30 21:01:41.197649 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.197558 5755 cpu_manager.go:414] "containerMap: removing container" podUID="73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5" containerName="revoke-keys" Oct 30 21:01:41.197649 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.197566 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5" containerName="revoke-keys" Oct 30 21:01:41.197844 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.197672 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd8a8354-e543-4fe8-8385-de8b3ea8b417" containerName="istio-proxy" Oct 30 21:01:41.197844 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.197684 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="41470888-65ed-44de-9bcc-210cf168dece" containerName="cleanup" Oct 30 21:01:41.197844 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.197700 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6b63f7-d73f-4b1a-8615-09d2f8ae2cc7" containerName="istio-proxy" Oct 30 21:01:41.197844 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.197710 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5" containerName="revoke-keys" Oct 30 21:01:41.200841 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.200819 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.246725 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.246686 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn"] Oct 30 21:01:41.250108 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.250085 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.255423 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:01:41.255380 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:01:41.286659 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.286616 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb613fb6-136d-49c9-a4dd-4ac76f320e96-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.286841 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.286708 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb613fb6-136d-49c9-a4dd-4ac76f320e96-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.286841 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.286771 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb613fb6-136d-49c9-a4dd-4ac76f320e96-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.286841 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.286796 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb613fb6-136d-49c9-a4dd-4ac76f320e96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.286967 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.286848 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb613fb6-136d-49c9-a4dd-4ac76f320e96-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.286967 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.286920 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb613fb6-136d-49c9-a4dd-4ac76f320e96-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.286967 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.286948 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb613fb6-136d-49c9-a4dd-4ac76f320e96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.287075 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.286967 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64cm9\" (UniqueName: \"kubernetes.io/projected/cb613fb6-136d-49c9-a4dd-4ac76f320e96-kube-api-access-64cm9\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.287075 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.287020 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb613fb6-136d-49c9-a4dd-4ac76f320e96-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.342771 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.342736 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn"] Oct 30 21:01:41.387876 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.387840 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb613fb6-136d-49c9-a4dd-4ac76f320e96-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.388083 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.387884 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb613fb6-136d-49c9-a4dd-4ac76f320e96-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.388083 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.387917 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4fdfa05-c965-4c76-a832-fa36a140952e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.388083 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.387949 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb613fb6-136d-49c9-a4dd-4ac76f320e96-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.388083 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388000 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4fdfa05-c965-4c76-a832-fa36a140952e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.388278 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388117 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb613fb6-136d-49c9-a4dd-4ac76f320e96-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.388278 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388153 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4fdfa05-c965-4c76-a832-fa36a140952e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.388278 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388176 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb613fb6-136d-49c9-a4dd-4ac76f320e96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.388278 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388242 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4fdfa05-c965-4c76-a832-fa36a140952e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.388278 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388273 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb613fb6-136d-49c9-a4dd-4ac76f320e96-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.388628 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388283 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb613fb6-136d-49c9-a4dd-4ac76f320e96-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.388628 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388325 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb613fb6-136d-49c9-a4dd-4ac76f320e96-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.388628 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388373 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4fdfa05-c965-4c76-a832-fa36a140952e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.388628 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388460 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4fdfa05-c965-4c76-a832-fa36a140952e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.388628 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388524 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb613fb6-136d-49c9-a4dd-4ac76f320e96-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.388628 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388546 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4fdfa05-c965-4c76-a832-fa36a140952e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.388628 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388562 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb613fb6-136d-49c9-a4dd-4ac76f320e96-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.388628 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388579 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb613fb6-136d-49c9-a4dd-4ac76f320e96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.388628 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388593 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb613fb6-136d-49c9-a4dd-4ac76f320e96-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.388628 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388606 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmlgk\" (UniqueName: \"kubernetes.io/projected/e4fdfa05-c965-4c76-a832-fa36a140952e-kube-api-access-zmlgk\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.389085 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388750 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4fdfa05-c965-4c76-a832-fa36a140952e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.389085 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388794 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64cm9\" (UniqueName: \"kubernetes.io/projected/cb613fb6-136d-49c9-a4dd-4ac76f320e96-kube-api-access-64cm9\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.389085 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.388859 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb613fb6-136d-49c9-a4dd-4ac76f320e96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.390744 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.390727 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb613fb6-136d-49c9-a4dd-4ac76f320e96-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.391115 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.391099 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb613fb6-136d-49c9-a4dd-4ac76f320e96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.400684 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.400664 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s"] Oct 30 21:01:41.446210 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.446171 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb613fb6-136d-49c9-a4dd-4ac76f320e96-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.446678 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.446652 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64cm9\" (UniqueName: \"kubernetes.io/projected/cb613fb6-136d-49c9-a4dd-4ac76f320e96-kube-api-access-64cm9\") pod \"maas-default-gateway-openshift-default-58847f9fc5-24b4s\" (UID: \"cb613fb6-136d-49c9-a4dd-4ac76f320e96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.489882 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.489844 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4fdfa05-c965-4c76-a832-fa36a140952e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.490064 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.489899 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4fdfa05-c965-4c76-a832-fa36a140952e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.490064 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.489926 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4fdfa05-c965-4c76-a832-fa36a140952e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.490064 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.489951 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4fdfa05-c965-4c76-a832-fa36a140952e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.490064 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.489970 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4fdfa05-c965-4c76-a832-fa36a140952e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.490064 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.489984 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4fdfa05-c965-4c76-a832-fa36a140952e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.490320 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.490077 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4fdfa05-c965-4c76-a832-fa36a140952e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.490320 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.490131 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmlgk\" (UniqueName: \"kubernetes.io/projected/e4fdfa05-c965-4c76-a832-fa36a140952e-kube-api-access-zmlgk\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.490320 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.490161 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4fdfa05-c965-4c76-a832-fa36a140952e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.490644 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.490374 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4fdfa05-c965-4c76-a832-fa36a140952e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.490644 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.490465 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4fdfa05-c965-4c76-a832-fa36a140952e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.490644 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.490533 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4fdfa05-c965-4c76-a832-fa36a140952e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.490644 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.490638 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4fdfa05-c965-4c76-a832-fa36a140952e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.490827 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.490696 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4fdfa05-c965-4c76-a832-fa36a140952e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.492410 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.492373 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4fdfa05-c965-4c76-a832-fa36a140952e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.492646 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.492630 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4fdfa05-c965-4c76-a832-fa36a140952e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.514134 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.514101 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:41.548338 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.548303 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4fdfa05-c965-4c76-a832-fa36a140952e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.548511 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.548423 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmlgk\" (UniqueName: \"kubernetes.io/projected/e4fdfa05-c965-4c76-a832-fa36a140952e-kube-api-access-zmlgk\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nttzn\" (UID: \"e4fdfa05-c965-4c76-a832-fa36a140952e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.563415 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.562915 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:41.941000 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.940909 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s"] Oct 30 21:01:41.978899 ip-10-0-139-195 kubenswrapper[5755]: W1030 21:01:41.978867 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb613fb6_136d_49c9_a4dd_4ac76f320e96.slice/crio-01f6d17b30eeaabea54e5f28fb96b7e1e8bcd1313d4283f8a603c4f90c81b9e4 WatchSource:0}: Error finding container 01f6d17b30eeaabea54e5f28fb96b7e1e8bcd1313d4283f8a603c4f90c81b9e4: Status 404 returned error can't find the container with id 01f6d17b30eeaabea54e5f28fb96b7e1e8bcd1313d4283f8a603c4f90c81b9e4 Oct 30 21:01:41.981082 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.981046 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:01:41.981207 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.981135 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:01:41.981207 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.981178 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:01:41.997605 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:41.994842 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn"] Oct 30 21:01:41.998850 ip-10-0-139-195 kubenswrapper[5755]: W1030 21:01:41.998824 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fdfa05_c965_4c76_a832_fa36a140952e.slice/crio-96b323409748468bf6c1c466625dad22061a616b48c8486a5f465968d9bdf8d9 WatchSource:0}: Error finding container 96b323409748468bf6c1c466625dad22061a616b48c8486a5f465968d9bdf8d9: Status 404 returned error can't find the container with id 96b323409748468bf6c1c466625dad22061a616b48c8486a5f465968d9bdf8d9 Oct 30 21:01:42.001060 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:42.001017 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:01:42.001138 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:42.001118 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:01:42.001176 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:42.001164 5755 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:01:42.485113 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:42.484150 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" event={"ID":"e4fdfa05-c965-4c76-a832-fa36a140952e","Type":"ContainerStarted","Data":"18a9bee2a87213b3eb673d7b791ed420522a98d7dc50944cec72a7dabfd8fdbe"} Oct 30 21:01:42.485113 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:42.484219 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" event={"ID":"e4fdfa05-c965-4c76-a832-fa36a140952e","Type":"ContainerStarted","Data":"96b323409748468bf6c1c466625dad22061a616b48c8486a5f465968d9bdf8d9"} Oct 30 21:01:42.488862 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:42.488145 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" event={"ID":"cb613fb6-136d-49c9-a4dd-4ac76f320e96","Type":"ContainerStarted","Data":"92e5b12ad107e46d25fc0932f067f41472e0c7ba5b45763bb2414869b16fc9ca"} Oct 30 21:01:42.488862 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:42.488206 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" event={"ID":"cb613fb6-136d-49c9-a4dd-4ac76f320e96","Type":"ContainerStarted","Data":"01f6d17b30eeaabea54e5f28fb96b7e1e8bcd1313d4283f8a603c4f90c81b9e4"} Oct 30 21:01:42.517618 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:42.517503 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:42.563497 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:42.563472 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:42.606073 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:42.605979 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" podStartSLOduration=1.6059589010000002 podStartE2EDuration="1.605958901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:42.60330952 +0000 UTC m=+2065.529866437" watchObservedRunningTime="2026-10-30 21:01:42.605958901 +0000 UTC m=+2065.532515816" Oct 30 21:01:42.725424 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:42.723823 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" podStartSLOduration=1.723802995 podStartE2EDuration="1.723802995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:42.721887294 +0000 UTC m=+2065.648444238" watchObservedRunningTime="2026-10-30 21:01:42.723802995 +0000 UTC m=+2065.650359911" Oct 30 21:01:43.516771 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:43.516433 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-24b4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:01:43.516771 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:43.516508 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" podUID="cb613fb6-136d-49c9-a4dd-4ac76f320e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 21:01:43.564432 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:43.564378 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nttzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:01:43.564627 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:43.564450 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" podUID="e4fdfa05-c965-4c76-a832-fa36a140952e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 21:01:44.518947 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:44.518899 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-24b4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:01:44.519625 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:44.519483 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" podUID="cb613fb6-136d-49c9-a4dd-4ac76f320e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 21:01:44.564168 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:44.564126 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nttzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:01:44.564357 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:44.564185 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" podUID="e4fdfa05-c965-4c76-a832-fa36a140952e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 21:01:45.519216 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:45.518774 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-24b4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:01:45.519216 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:45.518834 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" podUID="cb613fb6-136d-49c9-a4dd-4ac76f320e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 21:01:45.564359 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:45.564321 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nttzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:01:45.564554 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:45.564428 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" podUID="e4fdfa05-c965-4c76-a832-fa36a140952e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 21:01:46.517627 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:46.516802 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-24b4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:01:46.517627 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:46.516880 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" podUID="cb613fb6-136d-49c9-a4dd-4ac76f320e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 21:01:46.563360 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:46.563321 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nttzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:01:46.563768 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:46.563373 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" podUID="e4fdfa05-c965-4c76-a832-fa36a140952e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 21:01:47.520413 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:47.519881 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-24b4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:01:47.520413 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:47.519947 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" podUID="cb613fb6-136d-49c9-a4dd-4ac76f320e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 21:01:47.563475 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:47.563431 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nttzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:01:47.563954 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:47.563485 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" podUID="e4fdfa05-c965-4c76-a832-fa36a140952e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 21:01:48.515256 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:48.515211 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-24b4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:01:48.515467 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:48.515284 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" podUID="cb613fb6-136d-49c9-a4dd-4ac76f320e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 21:01:48.564157 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:48.564118 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nttzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:01:48.564589 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:48.564185 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" podUID="e4fdfa05-c965-4c76-a832-fa36a140952e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 21:01:49.515293 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:49.515250 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-24b4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:01:49.515497 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:49.515323 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" podUID="cb613fb6-136d-49c9-a4dd-4ac76f320e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 21:01:49.563721 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:49.563687 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nttzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:01:49.563894 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:49.563739 5755 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" podUID="e4fdfa05-c965-4c76-a832-fa36a140952e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 21:01:50.518628 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:50.518594 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:50.519121 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:50.518876 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:50.519706 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:50.519687 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" Oct 30 21:01:50.567469 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:50.567444 5755 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:50.567759 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:50.567740 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:01:50.568403 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:01:50.568370 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" Oct 30 21:02:11.255889 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:02:11.255813 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:02:18.316295 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:02:18.316262 5755 scope.go:122] "RemoveContainer" containerID="0aee27e23f26f33fb8b8fa01eea5f6775ea14ecfeacc9ed45c360966da15f165" Oct 30 21:02:41.255712 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:02:41.255665 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:03:11.255480 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:03:11.255444 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:03:18.347016 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:03:18.346968 5755 scope.go:122] "RemoveContainer" containerID="27d3b77051b01c62446a2d5a05a5412e1c09d21d98778036a0580ac3b7423818" Oct 30 21:03:18.356311 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:03:18.356286 5755 scope.go:122] "RemoveContainer" containerID="b4e553004cba8e970702cfaf40d2c8a014c3577938883782ebfbd119a5be94ab" Oct 30 21:03:18.364693 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:03:18.364673 5755 scope.go:122] "RemoveContainer" containerID="d7837dc4a1cdd068b2801a2077ac52f472e1fed469861e34c6faaec4153f274c" Oct 30 21:03:41.254913 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:03:41.254873 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:03:49.051725 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:03:49.051688 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp"] Oct 30 21:03:49.101594 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:03:49.101563 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s"] Oct 30 21:03:49.103454 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:03:49.103433 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f36ec720-f52dp"] Oct 30 21:03:49.104890 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:03:49.104866 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90960b22-nfh2s"] Oct 30 21:03:49.211120 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:03:49.211086 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18314524-5ec3-4f96-8a26-8696f8b1fbd3" path="/var/lib/kubelet/pods/18314524-5ec3-4f96-8a26-8696f8b1fbd3/volumes" Oct 30 21:03:49.211498 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:03:49.211484 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a12be28-f400-43ee-bb32-1c612e686542" path="/var/lib/kubelet/pods/3a12be28-f400-43ee-bb32-1c612e686542/volumes" Oct 30 21:03:55.055378 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:03:55.055343 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww"] Oct 30 21:03:55.102179 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:03:55.102146 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2"] Oct 30 21:03:55.103827 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:03:55.103802 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1781e14-pkpww"] Oct 30 21:03:55.106775 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:03:55.106756 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cacd0ea-b68s2"] Oct 30 21:03:55.211189 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:03:55.211152 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="499437e4-d0c8-4bb9-972f-9843b0233962" path="/var/lib/kubelet/pods/499437e4-d0c8-4bb9-972f-9843b0233962/volumes" Oct 30 21:03:55.211572 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:03:55.211558 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ad5871-dc94-4512-8f9b-3a754d439dca" path="/var/lib/kubelet/pods/54ad5871-dc94-4512-8f9b-3a754d439dca/volumes" Oct 30 21:04:06.562598 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:04:06.562555 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-24b4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:04:06.563010 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:04:06.562642 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" podUID="cb613fb6-136d-49c9-a4dd-4ac76f320e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:06.614422 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:04:06.610512 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nttzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:04:06.614422 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:04:06.610584 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" podUID="e4fdfa05-c965-4c76-a832-fa36a140952e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:11.255119 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:04:11.255080 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:04:18.397502 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:04:18.397468 5755 scope.go:122] "RemoveContainer" containerID="ea03a1cdf1612a04227d871bc6b1b205451595e1e3befa11e33d92ceb8f00904" Oct 30 21:04:18.406802 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:04:18.406779 5755 scope.go:122] "RemoveContainer" containerID="81ce4cc9b21194ac01ab1c4c3df852cf4f50731cc9ae7707f3285a991d8d3b10" Oct 30 21:04:18.414889 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:04:18.414870 5755 scope.go:122] "RemoveContainer" containerID="c9a2c2074bac0c18e01dbc7777f3eadded32065341d77396d5a4ae4ad3d6c128" Oct 30 21:04:41.051485 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:04:41.051447 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k"] Oct 30 21:04:41.101015 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:04:41.100975 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce6f2b33-gq47k"] Oct 30 21:04:41.102586 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:04:41.102553 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j"] Oct 30 21:04:41.103843 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:04:41.103825 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c641beaf-nqd9j"] Oct 30 21:04:41.210921 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:04:41.210886 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9392c8-52fd-48f2-af13-a7df13651950" path="/var/lib/kubelet/pods/0e9392c8-52fd-48f2-af13-a7df13651950/volumes" Oct 30 21:04:41.211287 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:04:41.211273 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f91abf06-f896-4a66-9653-1e7533769692" path="/var/lib/kubelet/pods/f91abf06-f896-4a66-9653-1e7533769692/volumes" Oct 30 21:04:41.254955 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:04:41.254922 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:05:11.050696 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:05:11.050607 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv"] Oct 30 21:05:11.100050 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:05:11.100015 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4950931-w7fjv"] Oct 30 21:05:11.211041 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:05:11.211003 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16840665-c3f7-48bf-a2bc-695c0be23df0" path="/var/lib/kubelet/pods/16840665-c3f7-48bf-a2bc-695c0be23df0/volumes" Oct 30 21:05:11.255311 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:05:11.255275 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:05:18.443963 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:05:18.443935 5755 scope.go:122] "RemoveContainer" containerID="d3548f27e5904ff70b18d55d868f95a0b714b82fab4c4012bacd9c59802ceaa1" Oct 30 21:05:18.452912 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:05:18.452885 5755 scope.go:122] "RemoveContainer" containerID="e0377bf7cf4e9ca612b6cb3f267411a3ec77e38ba6827113ae2676c0dbf19ae1" Oct 30 21:05:18.461006 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:05:18.460984 5755 scope.go:122] "RemoveContainer" containerID="2e69bc345a47e59699c054fe693a7d2daf35fe0fd38f4d0b74e18e34edc03aab" Oct 30 21:05:18.468576 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:05:18.468560 5755 scope.go:122] "RemoveContainer" containerID="491905521b5debbfa96138d72e8a46a1a14efe2a78d20bf0d201c0fb33cab3ce" Oct 30 21:05:18.476529 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:05:18.476510 5755 scope.go:122] "RemoveContainer" containerID="a6f8b190c38ce2401098777f72e24b261d6ad23f4a6ad2d815df6820db78c534" Oct 30 21:05:18.484238 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:05:18.484215 5755 scope.go:122] "RemoveContainer" containerID="b739334d4879597f4696d52197992845113a9592a8cb361a1f0bdd13ac61d848" Oct 30 21:05:18.491874 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:05:18.491852 5755 scope.go:122] "RemoveContainer" containerID="7b713b8eb95497abf842e177aa9821e851f4b921472b3cef32bc02126d398878" Oct 30 21:05:18.499764 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:05:18.499744 5755 scope.go:122] "RemoveContainer" containerID="2cd580dec563cf952dca5d8412a7c061e7ffe8acb15f840510a44b58ef17647b" Oct 30 21:05:41.254239 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:05:41.254202 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:06:00.051720 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:06:00.051674 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj"] Oct 30 21:06:00.107412 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:06:00.101583 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df662a7-hrlgj"] Oct 30 21:06:01.211227 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:06:01.211190 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5" path="/var/lib/kubelet/pods/73a6c2ff-fa1a-4b2c-9f80-d2a0757767e5/volumes" Oct 30 21:06:11.256021 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:06:11.255976 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:06:18.546884 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:06:18.546852 5755 scope.go:122] "RemoveContainer" containerID="63613f25fb4846b5557dbc939e0e0322f25837b2ee2c1e63391d1d7072066d88" Oct 30 21:06:18.556155 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:06:18.556130 5755 scope.go:122] "RemoveContainer" containerID="e3fd9fffbec67abc63f7224924322bf787a2eef20e5602ddd25678cd13a3ce3e" Oct 30 21:06:18.564498 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:06:18.564478 5755 scope.go:122] "RemoveContainer" containerID="68f7f8d803a89f51a447ccedb9dba3a6322b1cf2c41c9b7958dcc5aced4efbc3" Oct 30 21:06:41.255371 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:06:41.255332 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:07:11.254201 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:07:11.254164 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:07:18.590950 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:07:18.590916 5755 scope.go:122] "RemoveContainer" containerID="5d5df5a1f706dec22568d936b612f1d4bdd7b3e05a38ae5bd325fe10e7e0c6f7" Oct 30 21:07:41.255205 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:07:41.255170 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:08:11.255224 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:08:11.255135 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:08:21.568530 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:08:21.568489 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-24b4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:08:21.568969 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:08:21.568551 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-24b4s" podUID="cb613fb6-136d-49c9-a4dd-4ac76f320e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:08:21.615600 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:08:21.615527 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nttzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:08:21.615985 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:08:21.615947 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nttzn" podUID="e4fdfa05-c965-4c76-a832-fa36a140952e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:08:41.254041 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:08:41.254001 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:09:11.255853 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:09:11.255816 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:09:41.254551 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:09:41.254512 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:10:11.255406 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:10:11.255351 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:10:41.262226 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:10:41.262188 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:11:11.256510 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:11:11.256371 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:11:41.255271 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:11:41.255231 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:12:11.256205 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:12:11.256159 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:12:41.255105 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:12:41.255027 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:13:11.255883 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:13:11.255843 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:13:41.255308 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:13:41.255269 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:14:11.256309 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:14:11.256225 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:14:41.255881 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:14:41.255829 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:15:00.162891 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:00.162855 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz"] Oct 30 21:15:00.166305 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:00.166283 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" Oct 30 21:15:00.220665 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:00.220642 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\"" Oct 30 21:15:00.270948 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:00.270915 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz"] Oct 30 21:15:00.355309 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:00.355272 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e507777-ff19-40a4-8194-62d792c8ca8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-vbndz\" (UID: \"0e507777-ff19-40a4-8194-62d792c8ca8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" Oct 30 21:15:00.355518 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:00.355364 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2cjb\" (UniqueName: \"kubernetes.io/projected/0e507777-ff19-40a4-8194-62d792c8ca8a-kube-api-access-r2cjb\") pod \"maas-api-key-cleanup-29889915-vbndz\" (UID: \"0e507777-ff19-40a4-8194-62d792c8ca8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" Oct 30 21:15:00.456713 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:00.456626 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e507777-ff19-40a4-8194-62d792c8ca8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-vbndz\" (UID: \"0e507777-ff19-40a4-8194-62d792c8ca8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" Oct 30 21:15:00.456713 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:00.456709 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2cjb\" (UniqueName: \"kubernetes.io/projected/0e507777-ff19-40a4-8194-62d792c8ca8a-kube-api-access-r2cjb\") pod \"maas-api-key-cleanup-29889915-vbndz\" (UID: \"0e507777-ff19-40a4-8194-62d792c8ca8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" Oct 30 21:15:00.457344 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:00.457325 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e507777-ff19-40a4-8194-62d792c8ca8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-vbndz\" (UID: \"0e507777-ff19-40a4-8194-62d792c8ca8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" Oct 30 21:15:00.515421 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:00.515372 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2cjb\" (UniqueName: \"kubernetes.io/projected/0e507777-ff19-40a4-8194-62d792c8ca8a-kube-api-access-r2cjb\") pod \"maas-api-key-cleanup-29889915-vbndz\" (UID: \"0e507777-ff19-40a4-8194-62d792c8ca8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" Oct 30 21:15:00.778408 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:00.778351 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" Oct 30 21:15:01.150203 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:01.150179 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz"] Oct 30 21:15:01.152562 ip-10-0-139-195 kubenswrapper[5755]: W1030 21:15:01.152532 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e507777_ff19_40a4_8194_62d792c8ca8a.slice/crio-4c786a81b890f4f3fbccdce620b683b04eec1c75ba083e8d4a5834cce671301b WatchSource:0}: Error finding container 4c786a81b890f4f3fbccdce620b683b04eec1c75ba083e8d4a5834cce671301b: Status 404 returned error can't find the container with id 4c786a81b890f4f3fbccdce620b683b04eec1c75ba083e8d4a5834cce671301b Oct 30 21:15:01.772530 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:01.772424 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" event={"ID":"0e507777-ff19-40a4-8194-62d792c8ca8a","Type":"ContainerStarted","Data":"06d2a8d9147464bd0e95b7180b3b70b2e490238ec681b55c4d7d39102389672d"} Oct 30 21:15:01.772530 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:01.772493 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" event={"ID":"0e507777-ff19-40a4-8194-62d792c8ca8a","Type":"ContainerStarted","Data":"4c786a81b890f4f3fbccdce620b683b04eec1c75ba083e8d4a5834cce671301b"} Oct 30 21:15:01.887693 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:01.887567 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" podStartSLOduration=1.887553573 podStartE2EDuration="1.887553573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:01.886605209 +0000 UTC m=+2864.813162148" watchObservedRunningTime="2026-10-30 21:15:01.887553573 +0000 UTC m=+2864.814110484" Oct 30 21:15:02.779051 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:02.779018 5755 generic.go:356] "Generic (PLEG): container finished" podID="0e507777-ff19-40a4-8194-62d792c8ca8a" containerID="06d2a8d9147464bd0e95b7180b3b70b2e490238ec681b55c4d7d39102389672d" exitCode=60 Oct 30 21:15:02.779500 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:02.779082 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" event={"ID":"0e507777-ff19-40a4-8194-62d792c8ca8a","Type":"ContainerDied","Data":"06d2a8d9147464bd0e95b7180b3b70b2e490238ec681b55c4d7d39102389672d"} Oct 30 21:15:02.779500 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:02.779323 5755 scope.go:122] "RemoveContainer" containerID="06d2a8d9147464bd0e95b7180b3b70b2e490238ec681b55c4d7d39102389672d" Oct 30 21:15:03.786139 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:03.786103 5755 generic.go:356] "Generic (PLEG): container finished" podID="0e507777-ff19-40a4-8194-62d792c8ca8a" containerID="5837f5683c486423c3dd5bbb67bae9263b40da24e9f828f1abedb77064829aef" exitCode=60 Oct 30 21:15:03.786597 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:03.786153 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" event={"ID":"0e507777-ff19-40a4-8194-62d792c8ca8a","Type":"ContainerDied","Data":"5837f5683c486423c3dd5bbb67bae9263b40da24e9f828f1abedb77064829aef"} Oct 30 21:15:03.786597 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:03.786202 5755 scope.go:122] "RemoveContainer" containerID="06d2a8d9147464bd0e95b7180b3b70b2e490238ec681b55c4d7d39102389672d" Oct 30 21:15:03.786597 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:03.786465 5755 scope.go:122] "RemoveContainer" containerID="5837f5683c486423c3dd5bbb67bae9263b40da24e9f828f1abedb77064829aef" Oct 30 21:15:03.786765 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:15:03.786738 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-vbndz_odh-ai-gateway-infra(0e507777-ff19-40a4-8194-62d792c8ca8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" podUID="0e507777-ff19-40a4-8194-62d792c8ca8a" Oct 30 21:15:04.793144 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:04.793114 5755 scope.go:122] "RemoveContainer" containerID="5837f5683c486423c3dd5bbb67bae9263b40da24e9f828f1abedb77064829aef" Oct 30 21:15:04.793609 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:15:04.793336 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-vbndz_odh-ai-gateway-infra(0e507777-ff19-40a4-8194-62d792c8ca8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" podUID="0e507777-ff19-40a4-8194-62d792c8ca8a" Oct 30 21:15:11.255545 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:15:11.255502 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:15:17.207562 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:17.207524 5755 scope.go:122] "RemoveContainer" containerID="5837f5683c486423c3dd5bbb67bae9263b40da24e9f828f1abedb77064829aef" Oct 30 21:15:17.858592 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:17.858503 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" event={"ID":"0e507777-ff19-40a4-8194-62d792c8ca8a","Type":"ContainerStarted","Data":"b01c94a7476d2360173477c83b8a483467ca28d857a385cab46fdeb436edef3b"} Oct 30 21:15:18.865081 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:18.865045 5755 generic.go:356] "Generic (PLEG): container finished" podID="0e507777-ff19-40a4-8194-62d792c8ca8a" containerID="b01c94a7476d2360173477c83b8a483467ca28d857a385cab46fdeb436edef3b" exitCode=60 Oct 30 21:15:18.865543 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:18.865120 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" event={"ID":"0e507777-ff19-40a4-8194-62d792c8ca8a","Type":"ContainerDied","Data":"b01c94a7476d2360173477c83b8a483467ca28d857a385cab46fdeb436edef3b"} Oct 30 21:15:18.865543 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:18.865173 5755 scope.go:122] "RemoveContainer" containerID="5837f5683c486423c3dd5bbb67bae9263b40da24e9f828f1abedb77064829aef" Oct 30 21:15:18.865543 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:18.865530 5755 scope.go:122] "RemoveContainer" containerID="b01c94a7476d2360173477c83b8a483467ca28d857a385cab46fdeb436edef3b" Oct 30 21:15:18.865791 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:15:18.865773 5755 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-vbndz_odh-ai-gateway-infra(0e507777-ff19-40a4-8194-62d792c8ca8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" podUID="0e507777-ff19-40a4-8194-62d792c8ca8a" Oct 30 21:15:18.990548 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:18.990514 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz"] Oct 30 21:15:20.005977 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:20.005948 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" Oct 30 21:15:20.142604 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:20.142523 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e507777-ff19-40a4-8194-62d792c8ca8a-kube-api-access-r2cjb\" (UniqueName: \"kubernetes.io/projected/0e507777-ff19-40a4-8194-62d792c8ca8a-kube-api-access-r2cjb\") pod \"0e507777-ff19-40a4-8194-62d792c8ca8a\" (UID: \"0e507777-ff19-40a4-8194-62d792c8ca8a\") " Oct 30 21:15:20.142604 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:20.142564 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e507777-ff19-40a4-8194-62d792c8ca8a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e507777-ff19-40a4-8194-62d792c8ca8a-maas-api-service-ca\") pod \"0e507777-ff19-40a4-8194-62d792c8ca8a\" (UID: \"0e507777-ff19-40a4-8194-62d792c8ca8a\") " Oct 30 21:15:20.142971 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:20.142945 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e507777-ff19-40a4-8194-62d792c8ca8a-maas-api-service-ca" pod "0e507777-ff19-40a4-8194-62d792c8ca8a" (UID: "0e507777-ff19-40a4-8194-62d792c8ca8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:20.144811 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:20.144778 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e507777-ff19-40a4-8194-62d792c8ca8a-kube-api-access-r2cjb" pod "0e507777-ff19-40a4-8194-62d792c8ca8a" (UID: "0e507777-ff19-40a4-8194-62d792c8ca8a"). InnerVolumeSpecName "kube-api-access-r2cjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:20.244288 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:20.244257 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2cjb\" (UniqueName: \"kubernetes.io/projected/0e507777-ff19-40a4-8194-62d792c8ca8a-kube-api-access-r2cjb\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:20.244288 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:20.244284 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e507777-ff19-40a4-8194-62d792c8ca8a-maas-api-service-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:20.876891 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:20.876858 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" Oct 30 21:15:20.877107 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:20.876903 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz" event={"ID":"0e507777-ff19-40a4-8194-62d792c8ca8a","Type":"ContainerDied","Data":"4c786a81b890f4f3fbccdce620b683b04eec1c75ba083e8d4a5834cce671301b"} Oct 30 21:15:20.877107 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:20.876969 5755 scope.go:122] "RemoveContainer" containerID="b01c94a7476d2360173477c83b8a483467ca28d857a385cab46fdeb436edef3b" Oct 30 21:15:21.043558 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:21.043510 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz"] Oct 30 21:15:21.093213 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:21.093178 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vbndz"] Oct 30 21:15:21.212937 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:15:21.212859 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e507777-ff19-40a4-8194-62d792c8ca8a" path="/var/lib/kubelet/pods/0e507777-ff19-40a4-8194-62d792c8ca8a/volumes" Oct 30 21:15:41.256330 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:15:41.256246 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:16:11.254762 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:16:11.254719 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:16:41.254406 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:16:41.254351 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:17:01.533624 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:17:01.533509 5755 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-l95n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:17:01.533624 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:17:01.533601 5755 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l95n7" podUID="9ee1db28-bf77-435f-b92f-d6dbd06a5e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:17:11.255555 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:17:11.255518 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:17:41.255823 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:17:41.255781 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:18:11.255427 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:18:11.255370 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:18:41.255520 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:18:41.255442 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:19:07.080864 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.080826 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g"] Oct 30 21:19:07.081332 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.081213 5755 cpu_manager.go:414] "containerMap: removing container" podUID="0e507777-ff19-40a4-8194-62d792c8ca8a" containerName="cleanup" Oct 30 21:19:07.081332 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.081224 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e507777-ff19-40a4-8194-62d792c8ca8a" containerName="cleanup" Oct 30 21:19:07.081332 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.081244 5755 cpu_manager.go:414] "containerMap: removing container" podUID="0e507777-ff19-40a4-8194-62d792c8ca8a" containerName="cleanup" Oct 30 21:19:07.081332 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.081249 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e507777-ff19-40a4-8194-62d792c8ca8a" containerName="cleanup" Oct 30 21:19:07.081332 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.081305 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e507777-ff19-40a4-8194-62d792c8ca8a" containerName="cleanup" Oct 30 21:19:07.081332 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.081316 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e507777-ff19-40a4-8194-62d792c8ca8a" containerName="cleanup" Oct 30 21:19:07.081332 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.081322 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e507777-ff19-40a4-8194-62d792c8ca8a" containerName="cleanup" Oct 30 21:19:07.084413 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.084376 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" Oct 30 21:19:07.138179 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.138150 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:19:07.187047 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.187012 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lqclm\"" Oct 30 21:19:07.189046 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.189024 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g"] Oct 30 21:19:07.224708 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.224676 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b5dc20-f1b3-4f99-9742-303b0b5af71c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g\" (UID: \"32b5dc20-f1b3-4f99-9742-303b0b5af71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" Oct 30 21:19:07.224879 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.224737 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prtdk\" (UniqueName: \"kubernetes.io/projected/32b5dc20-f1b3-4f99-9742-303b0b5af71c-kube-api-access-prtdk\") pod \"maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g\" (UID: \"32b5dc20-f1b3-4f99-9742-303b0b5af71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" Oct 30 21:19:07.224879 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.224839 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b5dc20-f1b3-4f99-9742-303b0b5af71c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g\" (UID: \"32b5dc20-f1b3-4f99-9742-303b0b5af71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" Oct 30 21:19:07.325839 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.325796 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prtdk\" (UniqueName: \"kubernetes.io/projected/32b5dc20-f1b3-4f99-9742-303b0b5af71c-kube-api-access-prtdk\") pod \"maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g\" (UID: \"32b5dc20-f1b3-4f99-9742-303b0b5af71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" Oct 30 21:19:07.326047 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.325888 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b5dc20-f1b3-4f99-9742-303b0b5af71c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g\" (UID: \"32b5dc20-f1b3-4f99-9742-303b0b5af71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" Oct 30 21:19:07.326047 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.325947 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b5dc20-f1b3-4f99-9742-303b0b5af71c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g\" (UID: \"32b5dc20-f1b3-4f99-9742-303b0b5af71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" Oct 30 21:19:07.326593 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.326567 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b5dc20-f1b3-4f99-9742-303b0b5af71c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g\" (UID: \"32b5dc20-f1b3-4f99-9742-303b0b5af71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" Oct 30 21:19:07.328531 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.328507 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b5dc20-f1b3-4f99-9742-303b0b5af71c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g\" (UID: \"32b5dc20-f1b3-4f99-9742-303b0b5af71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" Oct 30 21:19:07.384507 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.384440 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prtdk\" (UniqueName: \"kubernetes.io/projected/32b5dc20-f1b3-4f99-9742-303b0b5af71c-kube-api-access-prtdk\") pod \"maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g\" (UID: \"32b5dc20-f1b3-4f99-9742-303b0b5af71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" Oct 30 21:19:07.397729 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.397700 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" Oct 30 21:19:07.767930 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:07.767907 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g"] Oct 30 21:19:07.770540 ip-10-0-139-195 kubenswrapper[5755]: W1030 21:19:07.770511 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b5dc20_f1b3_4f99_9742_303b0b5af71c.slice/crio-7064116e4af5efe4bd7f02ea46207641825aed8a5b968371be5d173aabe55504 WatchSource:0}: Error finding container 7064116e4af5efe4bd7f02ea46207641825aed8a5b968371be5d173aabe55504: Status 404 returned error can't find the container with id 7064116e4af5efe4bd7f02ea46207641825aed8a5b968371be5d173aabe55504 Oct 30 21:19:08.138485 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:08.138446 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" event={"ID":"32b5dc20-f1b3-4f99-9742-303b0b5af71c","Type":"ContainerStarted","Data":"5dc4d5177087a1ecb5464981dfd13a4540ff30ee7f879ec0a5f6cba4573d73aa"} Oct 30 21:19:08.138884 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:08.138502 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" event={"ID":"32b5dc20-f1b3-4f99-9742-303b0b5af71c","Type":"ContainerStarted","Data":"7064116e4af5efe4bd7f02ea46207641825aed8a5b968371be5d173aabe55504"} Oct 30 21:19:08.255049 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:08.254998 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" podStartSLOduration=1.254980974 podStartE2EDuration="1.254980974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:08.253835544 +0000 UTC m=+3111.180392482" watchObservedRunningTime="2026-10-30 21:19:08.254980974 +0000 UTC m=+3111.181537889" Oct 30 21:19:09.144541 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:09.144503 5755 generic.go:356] "Generic (PLEG): container finished" podID="32b5dc20-f1b3-4f99-9742-303b0b5af71c" containerID="5dc4d5177087a1ecb5464981dfd13a4540ff30ee7f879ec0a5f6cba4573d73aa" exitCode=0 Oct 30 21:19:09.144948 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:09.144594 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" event={"ID":"32b5dc20-f1b3-4f99-9742-303b0b5af71c","Type":"ContainerDied","Data":"5dc4d5177087a1ecb5464981dfd13a4540ff30ee7f879ec0a5f6cba4573d73aa"} Oct 30 21:19:10.287266 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:10.287235 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" Oct 30 21:19:10.455984 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:10.455903 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b5dc20-f1b3-4f99-9742-303b0b5af71c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b5dc20-f1b3-4f99-9742-303b0b5af71c-maas-api-openshift-ca\") pod \"32b5dc20-f1b3-4f99-9742-303b0b5af71c\" (UID: \"32b5dc20-f1b3-4f99-9742-303b0b5af71c\") " Oct 30 21:19:10.456169 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:10.455988 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32b5dc20-f1b3-4f99-9742-303b0b5af71c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b5dc20-f1b3-4f99-9742-303b0b5af71c-maas-api-xks-ca\") pod \"32b5dc20-f1b3-4f99-9742-303b0b5af71c\" (UID: \"32b5dc20-f1b3-4f99-9742-303b0b5af71c\") " Oct 30 21:19:10.456169 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:10.456135 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b5dc20-f1b3-4f99-9742-303b0b5af71c-kube-api-access-prtdk\" (UniqueName: \"kubernetes.io/projected/32b5dc20-f1b3-4f99-9742-303b0b5af71c-kube-api-access-prtdk\") pod \"32b5dc20-f1b3-4f99-9742-303b0b5af71c\" (UID: \"32b5dc20-f1b3-4f99-9742-303b0b5af71c\") " Oct 30 21:19:10.456322 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:10.456297 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b5dc20-f1b3-4f99-9742-303b0b5af71c-maas-api-openshift-ca" pod "32b5dc20-f1b3-4f99-9742-303b0b5af71c" (UID: "32b5dc20-f1b3-4f99-9742-303b0b5af71c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:10.456467 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:10.456447 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b5dc20-f1b3-4f99-9742-303b0b5af71c-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:10.458202 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:10.458182 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32b5dc20-f1b3-4f99-9742-303b0b5af71c-maas-api-xks-ca" pod "32b5dc20-f1b3-4f99-9742-303b0b5af71c" (UID: "32b5dc20-f1b3-4f99-9742-303b0b5af71c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:19:10.458403 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:10.458371 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b5dc20-f1b3-4f99-9742-303b0b5af71c-kube-api-access-prtdk" pod "32b5dc20-f1b3-4f99-9742-303b0b5af71c" (UID: "32b5dc20-f1b3-4f99-9742-303b0b5af71c"). InnerVolumeSpecName "kube-api-access-prtdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:10.557314 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:10.557274 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b5dc20-f1b3-4f99-9742-303b0b5af71c-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:10.557314 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:10.557307 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prtdk\" (UniqueName: \"kubernetes.io/projected/32b5dc20-f1b3-4f99-9742-303b0b5af71c-kube-api-access-prtdk\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:11.154817 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:11.154782 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" Oct 30 21:19:11.154817 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:11.154795 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g" event={"ID":"32b5dc20-f1b3-4f99-9742-303b0b5af71c","Type":"ContainerDied","Data":"7064116e4af5efe4bd7f02ea46207641825aed8a5b968371be5d173aabe55504"} Oct 30 21:19:11.155035 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:11.154839 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7064116e4af5efe4bd7f02ea46207641825aed8a5b968371be5d173aabe55504" Oct 30 21:19:11.255793 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:19:11.255759 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:19:26.726848 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:26.726806 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g"] Oct 30 21:19:26.776364 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:26.776327 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bddeea9c-z9t9g"] Oct 30 21:19:27.212304 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:19:27.212221 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b5dc20-f1b3-4f99-9742-303b0b5af71c" path="/var/lib/kubelet/pods/32b5dc20-f1b3-4f99-9742-303b0b5af71c/volumes" Oct 30 21:19:41.256116 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:19:41.256074 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:20:11.255225 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:20:11.255147 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:20:41.257680 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:20:41.257644 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:21:11.254660 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:21:11.254624 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:21:22.047569 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.047537 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk"] Oct 30 21:21:22.048079 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.047893 5755 cpu_manager.go:414] "containerMap: removing container" podUID="32b5dc20-f1b3-4f99-9742-303b0b5af71c" containerName="revoke-keys" Oct 30 21:21:22.048079 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.047903 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b5dc20-f1b3-4f99-9742-303b0b5af71c" containerName="revoke-keys" Oct 30 21:21:22.048079 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.047933 5755 cpu_manager.go:414] "containerMap: removing container" podUID="0e507777-ff19-40a4-8194-62d792c8ca8a" containerName="cleanup" Oct 30 21:21:22.048079 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.047938 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e507777-ff19-40a4-8194-62d792c8ca8a" containerName="cleanup" Oct 30 21:21:22.048079 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.048003 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b5dc20-f1b3-4f99-9742-303b0b5af71c" containerName="revoke-keys" Oct 30 21:21:22.051158 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.051143 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk" Oct 30 21:21:22.108473 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.108441 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk"] Oct 30 21:21:22.155860 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.155822 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:21:22.156067 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.155917 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lqclm\"" Oct 30 21:21:22.179895 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.179858 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk\" (UID: \"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk" Oct 30 21:21:22.180072 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.179913 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgchn\" (UniqueName: \"kubernetes.io/projected/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-kube-api-access-wgchn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk\" (UID: \"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk" Oct 30 21:21:22.180072 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.180027 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk\" (UID: \"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk" Oct 30 21:21:22.281022 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.280976 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk\" (UID: \"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk" Oct 30 21:21:22.281216 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.281052 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgchn\" (UniqueName: \"kubernetes.io/projected/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-kube-api-access-wgchn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk\" (UID: \"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk" Oct 30 21:21:22.281216 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.281094 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk\" (UID: \"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk" Oct 30 21:21:22.281865 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.281846 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk\" (UID: \"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk" Oct 30 21:21:22.283626 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.283602 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk\" (UID: \"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk" Oct 30 21:21:22.340830 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.340750 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgchn\" (UniqueName: \"kubernetes.io/projected/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-kube-api-access-wgchn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk\" (UID: \"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk" Oct 30 21:21:22.363221 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.363183 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk" Oct 30 21:21:22.733303 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.733277 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk"] Oct 30 21:21:22.736082 ip-10-0-139-195 kubenswrapper[5755]: W1030 21:21:22.736054 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ce11a2_94a4_4d3f_87f3_7a1858fbf17a.slice/crio-95cd433c72393ad3d8c0da66ad19b9b23b672a119f5cc2890004ada29de212ae WatchSource:0}: Error finding container 95cd433c72393ad3d8c0da66ad19b9b23b672a119f5cc2890004ada29de212ae: Status 404 returned error can't find the container with id 95cd433c72393ad3d8c0da66ad19b9b23b672a119f5cc2890004ada29de212ae Oct 30 21:21:22.786244 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:22.786210 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk" event={"ID":"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a","Type":"ContainerStarted","Data":"95cd433c72393ad3d8c0da66ad19b9b23b672a119f5cc2890004ada29de212ae"} Oct 30 21:21:23.792955 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:23.792910 5755 generic.go:356] "Generic (PLEG): container finished" podID="20ce11a2-94a4-4d3f-87f3-7a1858fbf17a" containerID="06a6f6d8de0e5ef057d23a4103e340d8a884d312bbf6fc00b7048acaa8ab40fd" exitCode=0 Oct 30 21:21:23.793332 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:23.792958 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk" event={"ID":"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a","Type":"ContainerDied","Data":"06a6f6d8de0e5ef057d23a4103e340d8a884d312bbf6fc00b7048acaa8ab40fd"} Oct 30 21:21:24.925020 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:24.924995 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk" Oct 30 21:21:25.109808 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:25.109726 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-maas-api-xks-ca\") pod \"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a\" (UID: \"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a\") " Oct 30 21:21:25.109970 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:25.109809 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-maas-api-openshift-ca\") pod \"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a\" (UID: \"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a\") " Oct 30 21:21:25.109970 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:25.109845 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-kube-api-access-wgchn\" (UniqueName: \"kubernetes.io/projected/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-kube-api-access-wgchn\") pod \"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a\" (UID: \"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a\") " Oct 30 21:21:25.110226 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:25.110196 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-maas-api-openshift-ca" pod "20ce11a2-94a4-4d3f-87f3-7a1858fbf17a" (UID: "20ce11a2-94a4-4d3f-87f3-7a1858fbf17a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:21:25.112058 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:25.112030 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-kube-api-access-wgchn" pod "20ce11a2-94a4-4d3f-87f3-7a1858fbf17a" (UID: "20ce11a2-94a4-4d3f-87f3-7a1858fbf17a"). InnerVolumeSpecName "kube-api-access-wgchn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:21:25.112151 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:25.112053 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-maas-api-xks-ca" pod "20ce11a2-94a4-4d3f-87f3-7a1858fbf17a" (UID: "20ce11a2-94a4-4d3f-87f3-7a1858fbf17a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:21:25.211442 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:25.211411 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:21:25.211442 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:25.211436 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:21:25.211442 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:25.211446 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgchn\" (UniqueName: \"kubernetes.io/projected/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a-kube-api-access-wgchn\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:21:25.804216 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:25.804129 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk" Oct 30 21:21:25.804216 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:25.804151 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk" event={"ID":"20ce11a2-94a4-4d3f-87f3-7a1858fbf17a","Type":"ContainerDied","Data":"95cd433c72393ad3d8c0da66ad19b9b23b672a119f5cc2890004ada29de212ae"} Oct 30 21:21:25.804216 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:25.804209 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95cd433c72393ad3d8c0da66ad19b9b23b672a119f5cc2890004ada29de212ae" Oct 30 21:21:41.254694 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:21:41.254583 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:21:45.498351 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:45.498319 5755 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk"] Oct 30 21:21:45.547970 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:45.547941 5755 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-545519-cg5jk"] Oct 30 21:21:47.211041 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:21:47.211006 5755 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ce11a2-94a4-4d3f-87f3-7a1858fbf17a" path="/var/lib/kubelet/pods/20ce11a2-94a4-4d3f-87f3-7a1858fbf17a/volumes" Oct 30 21:22:11.254985 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:22:11.254943 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:22:23.293221 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.293191 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm"] Oct 30 21:22:23.293745 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.293603 5755 cpu_manager.go:414] "containerMap: removing container" podUID="20ce11a2-94a4-4d3f-87f3-7a1858fbf17a" containerName="revoke-keys" Oct 30 21:22:23.293745 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.293617 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ce11a2-94a4-4d3f-87f3-7a1858fbf17a" containerName="revoke-keys" Oct 30 21:22:23.293745 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.293703 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ce11a2-94a4-4d3f-87f3-7a1858fbf17a" containerName="revoke-keys" Oct 30 21:22:23.296757 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.296742 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm" Oct 30 21:22:23.353767 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.353734 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm"] Oct 30 21:22:23.390292 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.390259 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9rsz\" (UniqueName: \"kubernetes.io/projected/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-kube-api-access-v9rsz\") pod \"maas-api-revoke-sub-f03629db-s4vtm\" (UID: \"007a48c5-b3be-4b1b-83d8-cc5a03364fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm" Oct 30 21:22:23.390477 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.390322 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-s4vtm\" (UID: \"007a48c5-b3be-4b1b-83d8-cc5a03364fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm" Oct 30 21:22:23.390477 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.390445 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-s4vtm\" (UID: \"007a48c5-b3be-4b1b-83d8-cc5a03364fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm" Oct 30 21:22:23.401315 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.401286 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lqclm\"" Oct 30 21:22:23.401315 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.401299 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:22:23.490920 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.490890 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9rsz\" (UniqueName: \"kubernetes.io/projected/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-kube-api-access-v9rsz\") pod \"maas-api-revoke-sub-f03629db-s4vtm\" (UID: \"007a48c5-b3be-4b1b-83d8-cc5a03364fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm" Oct 30 21:22:23.491118 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.490954 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-s4vtm\" (UID: \"007a48c5-b3be-4b1b-83d8-cc5a03364fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm" Oct 30 21:22:23.491118 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.490985 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-s4vtm\" (UID: \"007a48c5-b3be-4b1b-83d8-cc5a03364fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm" Oct 30 21:22:23.491611 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.491588 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-s4vtm\" (UID: \"007a48c5-b3be-4b1b-83d8-cc5a03364fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm" Oct 30 21:22:23.493455 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.493425 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-s4vtm\" (UID: \"007a48c5-b3be-4b1b-83d8-cc5a03364fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm" Oct 30 21:22:23.550115 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.550048 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9rsz\" (UniqueName: \"kubernetes.io/projected/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-kube-api-access-v9rsz\") pod \"maas-api-revoke-sub-f03629db-s4vtm\" (UID: \"007a48c5-b3be-4b1b-83d8-cc5a03364fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm" Oct 30 21:22:23.609276 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.609244 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm" Oct 30 21:22:23.983013 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:23.982987 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm"] Oct 30 21:22:23.984931 ip-10-0-139-195 kubenswrapper[5755]: W1030 21:22:23.984904 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007a48c5_b3be_4b1b_83d8_cc5a03364fc3.slice/crio-f1c3ef139db0122c519b9cba914acffc67f83a268e7ec5aaddae11ae88b7d31a WatchSource:0}: Error finding container f1c3ef139db0122c519b9cba914acffc67f83a268e7ec5aaddae11ae88b7d31a: Status 404 returned error can't find the container with id f1c3ef139db0122c519b9cba914acffc67f83a268e7ec5aaddae11ae88b7d31a Oct 30 21:22:24.089565 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:24.089533 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm" event={"ID":"007a48c5-b3be-4b1b-83d8-cc5a03364fc3","Type":"ContainerStarted","Data":"f1c3ef139db0122c519b9cba914acffc67f83a268e7ec5aaddae11ae88b7d31a"} Oct 30 21:22:25.095805 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:25.095710 5755 generic.go:356] "Generic (PLEG): container finished" podID="007a48c5-b3be-4b1b-83d8-cc5a03364fc3" containerID="081990df76bce29dab9cb5204058d10ede9cb310efd370374eadb1a9220e1678" exitCode=0 Oct 30 21:22:25.095805 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:25.095773 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm" event={"ID":"007a48c5-b3be-4b1b-83d8-cc5a03364fc3","Type":"ContainerDied","Data":"081990df76bce29dab9cb5204058d10ede9cb310efd370374eadb1a9220e1678"} Oct 30 21:22:26.229803 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:26.229779 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm" Oct 30 21:22:26.311125 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:26.311092 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-kube-api-access-v9rsz\" (UniqueName: \"kubernetes.io/projected/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-kube-api-access-v9rsz\") pod \"007a48c5-b3be-4b1b-83d8-cc5a03364fc3\" (UID: \"007a48c5-b3be-4b1b-83d8-cc5a03364fc3\") " Oct 30 21:22:26.311323 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:26.311152 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-maas-api-xks-ca\") pod \"007a48c5-b3be-4b1b-83d8-cc5a03364fc3\" (UID: \"007a48c5-b3be-4b1b-83d8-cc5a03364fc3\") " Oct 30 21:22:26.311323 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:26.311173 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-maas-api-openshift-ca\") pod \"007a48c5-b3be-4b1b-83d8-cc5a03364fc3\" (UID: \"007a48c5-b3be-4b1b-83d8-cc5a03364fc3\") " Oct 30 21:22:26.311604 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:26.311580 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-maas-api-openshift-ca" pod "007a48c5-b3be-4b1b-83d8-cc5a03364fc3" (UID: "007a48c5-b3be-4b1b-83d8-cc5a03364fc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:22:26.313358 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:26.313336 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-kube-api-access-v9rsz" pod "007a48c5-b3be-4b1b-83d8-cc5a03364fc3" (UID: "007a48c5-b3be-4b1b-83d8-cc5a03364fc3"). InnerVolumeSpecName "kube-api-access-v9rsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:22:26.313358 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:26.313346 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-maas-api-xks-ca" pod "007a48c5-b3be-4b1b-83d8-cc5a03364fc3" (UID: "007a48c5-b3be-4b1b-83d8-cc5a03364fc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:22:26.412565 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:26.412487 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9rsz\" (UniqueName: \"kubernetes.io/projected/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-kube-api-access-v9rsz\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:26.412565 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:26.412514 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:26.412565 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:26.412523 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007a48c5-b3be-4b1b-83d8-cc5a03364fc3-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:27.105711 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:27.105675 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm" event={"ID":"007a48c5-b3be-4b1b-83d8-cc5a03364fc3","Type":"ContainerDied","Data":"f1c3ef139db0122c519b9cba914acffc67f83a268e7ec5aaddae11ae88b7d31a"} Oct 30 21:22:27.105888 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:27.105725 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1c3ef139db0122c519b9cba914acffc67f83a268e7ec5aaddae11ae88b7d31a" Oct 30 21:22:27.105888 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:27.105689 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s4vtm" Oct 30 21:22:41.262284 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:22:41.262223 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:22:43.997133 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:43.997095 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt"] Oct 30 21:22:43.997702 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:43.997682 5755 cpu_manager.go:414] "containerMap: removing container" podUID="007a48c5-b3be-4b1b-83d8-cc5a03364fc3" containerName="revoke-keys" Oct 30 21:22:43.997780 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:43.997705 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007a48c5-b3be-4b1b-83d8-cc5a03364fc3" containerName="revoke-keys" Oct 30 21:22:43.997836 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:43.997817 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="007a48c5-b3be-4b1b-83d8-cc5a03364fc3" containerName="revoke-keys" Oct 30 21:22:44.000948 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:44.000926 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" Oct 30 21:22:44.060548 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:44.060517 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt"] Oct 30 21:22:44.085426 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:44.085373 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nslp\" (UniqueName: \"kubernetes.io/projected/9c511f0f-6db4-47ac-94e1-c4e831d5f858-kube-api-access-6nslp\") pod \"maas-api-revoke-sub-0165d7d2-5ddrt\" (UID: \"9c511f0f-6db4-47ac-94e1-c4e831d5f858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" Oct 30 21:22:44.085625 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:44.085455 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c511f0f-6db4-47ac-94e1-c4e831d5f858-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-5ddrt\" (UID: \"9c511f0f-6db4-47ac-94e1-c4e831d5f858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" Oct 30 21:22:44.085625 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:44.085592 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c511f0f-6db4-47ac-94e1-c4e831d5f858-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-5ddrt\" (UID: \"9c511f0f-6db4-47ac-94e1-c4e831d5f858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" Oct 30 21:22:44.107769 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:44.107735 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:22:44.107923 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:44.107818 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lqclm\"" Oct 30 21:22:44.186273 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:44.186239 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nslp\" (UniqueName: \"kubernetes.io/projected/9c511f0f-6db4-47ac-94e1-c4e831d5f858-kube-api-access-6nslp\") pod \"maas-api-revoke-sub-0165d7d2-5ddrt\" (UID: \"9c511f0f-6db4-47ac-94e1-c4e831d5f858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" Oct 30 21:22:44.186468 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:44.186288 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c511f0f-6db4-47ac-94e1-c4e831d5f858-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-5ddrt\" (UID: \"9c511f0f-6db4-47ac-94e1-c4e831d5f858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" Oct 30 21:22:44.186468 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:44.186348 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c511f0f-6db4-47ac-94e1-c4e831d5f858-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-5ddrt\" (UID: \"9c511f0f-6db4-47ac-94e1-c4e831d5f858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" Oct 30 21:22:44.187009 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:44.186978 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c511f0f-6db4-47ac-94e1-c4e831d5f858-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-5ddrt\" (UID: \"9c511f0f-6db4-47ac-94e1-c4e831d5f858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" Oct 30 21:22:44.188826 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:44.188808 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c511f0f-6db4-47ac-94e1-c4e831d5f858-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-5ddrt\" (UID: \"9c511f0f-6db4-47ac-94e1-c4e831d5f858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" Oct 30 21:22:44.244588 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:44.244553 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nslp\" (UniqueName: \"kubernetes.io/projected/9c511f0f-6db4-47ac-94e1-c4e831d5f858-kube-api-access-6nslp\") pod \"maas-api-revoke-sub-0165d7d2-5ddrt\" (UID: \"9c511f0f-6db4-47ac-94e1-c4e831d5f858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" Oct 30 21:22:44.316155 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:44.316067 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" Oct 30 21:22:44.691451 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:44.691423 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt"] Oct 30 21:22:44.693026 ip-10-0-139-195 kubenswrapper[5755]: W1030 21:22:44.692996 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c511f0f_6db4_47ac_94e1_c4e831d5f858.slice/crio-964accf078b99f170a6a8d6ef515f2f54a7e53548d0c34e95afb210326abe66f WatchSource:0}: Error finding container 964accf078b99f170a6a8d6ef515f2f54a7e53548d0c34e95afb210326abe66f: Status 404 returned error can't find the container with id 964accf078b99f170a6a8d6ef515f2f54a7e53548d0c34e95afb210326abe66f Oct 30 21:22:45.232506 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:45.232418 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" event={"ID":"9c511f0f-6db4-47ac-94e1-c4e831d5f858","Type":"ContainerStarted","Data":"99a89acddbc09723054f8604a5cb5d81d2caa1a80a7e13352820907ed77f94b5"} Oct 30 21:22:45.232876 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:45.232522 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" event={"ID":"9c511f0f-6db4-47ac-94e1-c4e831d5f858","Type":"ContainerStarted","Data":"964accf078b99f170a6a8d6ef515f2f54a7e53548d0c34e95afb210326abe66f"} Oct 30 21:22:45.351863 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:45.351812 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" podStartSLOduration=2.351793969 podStartE2EDuration="2.351793969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:45.349930276 +0000 UTC m=+3328.276487191" watchObservedRunningTime="2026-10-30 21:22:45.351793969 +0000 UTC m=+3328.278350893" Oct 30 21:22:46.239359 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:46.239324 5755 generic.go:356] "Generic (PLEG): container finished" podID="9c511f0f-6db4-47ac-94e1-c4e831d5f858" containerID="99a89acddbc09723054f8604a5cb5d81d2caa1a80a7e13352820907ed77f94b5" exitCode=0 Oct 30 21:22:46.239813 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:46.239414 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" event={"ID":"9c511f0f-6db4-47ac-94e1-c4e831d5f858","Type":"ContainerDied","Data":"99a89acddbc09723054f8604a5cb5d81d2caa1a80a7e13352820907ed77f94b5"} Oct 30 21:22:47.377592 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:47.377563 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" Oct 30 21:22:47.517881 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:47.517803 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c511f0f-6db4-47ac-94e1-c4e831d5f858-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c511f0f-6db4-47ac-94e1-c4e831d5f858-maas-api-xks-ca\") pod \"9c511f0f-6db4-47ac-94e1-c4e831d5f858\" (UID: \"9c511f0f-6db4-47ac-94e1-c4e831d5f858\") " Oct 30 21:22:47.517881 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:47.517865 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c511f0f-6db4-47ac-94e1-c4e831d5f858-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c511f0f-6db4-47ac-94e1-c4e831d5f858-maas-api-openshift-ca\") pod \"9c511f0f-6db4-47ac-94e1-c4e831d5f858\" (UID: \"9c511f0f-6db4-47ac-94e1-c4e831d5f858\") " Oct 30 21:22:47.518098 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:47.517907 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c511f0f-6db4-47ac-94e1-c4e831d5f858-kube-api-access-6nslp\" (UniqueName: \"kubernetes.io/projected/9c511f0f-6db4-47ac-94e1-c4e831d5f858-kube-api-access-6nslp\") pod \"9c511f0f-6db4-47ac-94e1-c4e831d5f858\" (UID: \"9c511f0f-6db4-47ac-94e1-c4e831d5f858\") " Oct 30 21:22:47.518354 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:47.518318 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c511f0f-6db4-47ac-94e1-c4e831d5f858-maas-api-openshift-ca" pod "9c511f0f-6db4-47ac-94e1-c4e831d5f858" (UID: "9c511f0f-6db4-47ac-94e1-c4e831d5f858"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:22:47.520176 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:47.520138 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c511f0f-6db4-47ac-94e1-c4e831d5f858-maas-api-xks-ca" pod "9c511f0f-6db4-47ac-94e1-c4e831d5f858" (UID: "9c511f0f-6db4-47ac-94e1-c4e831d5f858"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:22:47.520272 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:47.520196 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c511f0f-6db4-47ac-94e1-c4e831d5f858-kube-api-access-6nslp" pod "9c511f0f-6db4-47ac-94e1-c4e831d5f858" (UID: "9c511f0f-6db4-47ac-94e1-c4e831d5f858"). InnerVolumeSpecName "kube-api-access-6nslp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:22:47.619453 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:47.619407 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c511f0f-6db4-47ac-94e1-c4e831d5f858-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:47.619453 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:47.619452 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c511f0f-6db4-47ac-94e1-c4e831d5f858-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:47.619453 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:47.619462 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nslp\" (UniqueName: \"kubernetes.io/projected/9c511f0f-6db4-47ac-94e1-c4e831d5f858-kube-api-access-6nslp\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:48.251209 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:48.251179 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" Oct 30 21:22:48.251209 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:48.251188 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5ddrt" event={"ID":"9c511f0f-6db4-47ac-94e1-c4e831d5f858","Type":"ContainerDied","Data":"964accf078b99f170a6a8d6ef515f2f54a7e53548d0c34e95afb210326abe66f"} Oct 30 21:22:48.251445 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:48.251232 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="964accf078b99f170a6a8d6ef515f2f54a7e53548d0c34e95afb210326abe66f" Oct 30 21:22:55.834806 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:55.834721 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk"] Oct 30 21:22:55.835202 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:55.835124 5755 cpu_manager.go:414] "containerMap: removing container" podUID="9c511f0f-6db4-47ac-94e1-c4e831d5f858" containerName="revoke-keys" Oct 30 21:22:55.835202 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:55.835138 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c511f0f-6db4-47ac-94e1-c4e831d5f858" containerName="revoke-keys" Oct 30 21:22:55.835292 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:55.835225 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c511f0f-6db4-47ac-94e1-c4e831d5f858" containerName="revoke-keys" Oct 30 21:22:55.838295 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:55.838278 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" Oct 30 21:22:55.895362 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:55.895327 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk"] Oct 30 21:22:55.942427 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:55.942379 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:22:55.942427 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:55.942405 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lqclm\"" Oct 30 21:22:56.000739 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:56.000702 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp7fn\" (UniqueName: \"kubernetes.io/projected/ba0b956f-3707-44d7-84b0-1b05d3162198-kube-api-access-dp7fn\") pod \"maas-api-revoke-sub-6b19338f-v5tvk\" (UID: \"ba0b956f-3707-44d7-84b0-1b05d3162198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" Oct 30 21:22:56.000922 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:56.000749 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0b956f-3707-44d7-84b0-1b05d3162198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-v5tvk\" (UID: \"ba0b956f-3707-44d7-84b0-1b05d3162198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" Oct 30 21:22:56.000922 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:56.000873 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0b956f-3707-44d7-84b0-1b05d3162198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-v5tvk\" (UID: \"ba0b956f-3707-44d7-84b0-1b05d3162198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" Oct 30 21:22:56.101478 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:56.101398 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0b956f-3707-44d7-84b0-1b05d3162198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-v5tvk\" (UID: \"ba0b956f-3707-44d7-84b0-1b05d3162198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" Oct 30 21:22:56.101636 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:56.101497 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp7fn\" (UniqueName: \"kubernetes.io/projected/ba0b956f-3707-44d7-84b0-1b05d3162198-kube-api-access-dp7fn\") pod \"maas-api-revoke-sub-6b19338f-v5tvk\" (UID: \"ba0b956f-3707-44d7-84b0-1b05d3162198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" Oct 30 21:22:56.101636 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:56.101523 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0b956f-3707-44d7-84b0-1b05d3162198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-v5tvk\" (UID: \"ba0b956f-3707-44d7-84b0-1b05d3162198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" Oct 30 21:22:56.102145 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:56.102127 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0b956f-3707-44d7-84b0-1b05d3162198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-v5tvk\" (UID: \"ba0b956f-3707-44d7-84b0-1b05d3162198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" Oct 30 21:22:56.103950 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:56.103929 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0b956f-3707-44d7-84b0-1b05d3162198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-v5tvk\" (UID: \"ba0b956f-3707-44d7-84b0-1b05d3162198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" Oct 30 21:22:56.160805 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:56.160770 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp7fn\" (UniqueName: \"kubernetes.io/projected/ba0b956f-3707-44d7-84b0-1b05d3162198-kube-api-access-dp7fn\") pod \"maas-api-revoke-sub-6b19338f-v5tvk\" (UID: \"ba0b956f-3707-44d7-84b0-1b05d3162198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" Oct 30 21:22:56.450494 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:56.450410 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" Oct 30 21:22:56.822744 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:56.822716 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk"] Oct 30 21:22:56.824874 ip-10-0-139-195 kubenswrapper[5755]: W1030 21:22:56.824845 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0b956f_3707_44d7_84b0_1b05d3162198.slice/crio-21484dc7dd1d1abef8596e5a8fae3ab36a4fd2295d7ba1bde5a57185e19c2456 WatchSource:0}: Error finding container 21484dc7dd1d1abef8596e5a8fae3ab36a4fd2295d7ba1bde5a57185e19c2456: Status 404 returned error can't find the container with id 21484dc7dd1d1abef8596e5a8fae3ab36a4fd2295d7ba1bde5a57185e19c2456 Oct 30 21:22:57.298430 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:57.298366 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" event={"ID":"ba0b956f-3707-44d7-84b0-1b05d3162198","Type":"ContainerStarted","Data":"94e64f753524c2e0295ca866e8b33415dbba4d4373253b20a45d381d564daec3"} Oct 30 21:22:57.298845 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:57.298440 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" event={"ID":"ba0b956f-3707-44d7-84b0-1b05d3162198","Type":"ContainerStarted","Data":"21484dc7dd1d1abef8596e5a8fae3ab36a4fd2295d7ba1bde5a57185e19c2456"} Oct 30 21:22:57.427688 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:57.427582 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" podStartSLOduration=2.427567937 podStartE2EDuration="2.427567937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:57.425713791 +0000 UTC m=+3340.352270723" watchObservedRunningTime="2026-10-30 21:22:57.427567937 +0000 UTC m=+3340.354124851" Oct 30 21:22:58.305202 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:58.305166 5755 generic.go:356] "Generic (PLEG): container finished" podID="ba0b956f-3707-44d7-84b0-1b05d3162198" containerID="94e64f753524c2e0295ca866e8b33415dbba4d4373253b20a45d381d564daec3" exitCode=0 Oct 30 21:22:58.305683 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:58.305256 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" event={"ID":"ba0b956f-3707-44d7-84b0-1b05d3162198","Type":"ContainerDied","Data":"94e64f753524c2e0295ca866e8b33415dbba4d4373253b20a45d381d564daec3"} Oct 30 21:22:59.448084 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:59.448060 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" Oct 30 21:22:59.535582 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:59.535545 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba0b956f-3707-44d7-84b0-1b05d3162198-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0b956f-3707-44d7-84b0-1b05d3162198-maas-api-xks-ca\") pod \"ba0b956f-3707-44d7-84b0-1b05d3162198\" (UID: \"ba0b956f-3707-44d7-84b0-1b05d3162198\") " Oct 30 21:22:59.535795 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:59.535592 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0b956f-3707-44d7-84b0-1b05d3162198-kube-api-access-dp7fn\" (UniqueName: \"kubernetes.io/projected/ba0b956f-3707-44d7-84b0-1b05d3162198-kube-api-access-dp7fn\") pod \"ba0b956f-3707-44d7-84b0-1b05d3162198\" (UID: \"ba0b956f-3707-44d7-84b0-1b05d3162198\") " Oct 30 21:22:59.535795 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:59.535731 5755 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0b956f-3707-44d7-84b0-1b05d3162198-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0b956f-3707-44d7-84b0-1b05d3162198-maas-api-openshift-ca\") pod \"ba0b956f-3707-44d7-84b0-1b05d3162198\" (UID: \"ba0b956f-3707-44d7-84b0-1b05d3162198\") " Oct 30 21:22:59.536012 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:59.535992 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0b956f-3707-44d7-84b0-1b05d3162198-maas-api-openshift-ca" pod "ba0b956f-3707-44d7-84b0-1b05d3162198" (UID: "ba0b956f-3707-44d7-84b0-1b05d3162198"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:22:59.537843 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:59.537822 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0b956f-3707-44d7-84b0-1b05d3162198-kube-api-access-dp7fn" pod "ba0b956f-3707-44d7-84b0-1b05d3162198" (UID: "ba0b956f-3707-44d7-84b0-1b05d3162198"). InnerVolumeSpecName "kube-api-access-dp7fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:22:59.537934 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:59.537882 5755 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba0b956f-3707-44d7-84b0-1b05d3162198-maas-api-xks-ca" pod "ba0b956f-3707-44d7-84b0-1b05d3162198" (UID: "ba0b956f-3707-44d7-84b0-1b05d3162198"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:22:59.636849 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:59.636750 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0b956f-3707-44d7-84b0-1b05d3162198-maas-api-openshift-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:59.636849 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:59.636789 5755 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0b956f-3707-44d7-84b0-1b05d3162198-maas-api-xks-ca\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:59.636849 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:22:59.636803 5755 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp7fn\" (UniqueName: \"kubernetes.io/projected/ba0b956f-3707-44d7-84b0-1b05d3162198-kube-api-access-dp7fn\") on node \"ip-10-0-139-195.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:00.316708 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:00.316668 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" event={"ID":"ba0b956f-3707-44d7-84b0-1b05d3162198","Type":"ContainerDied","Data":"21484dc7dd1d1abef8596e5a8fae3ab36a4fd2295d7ba1bde5a57185e19c2456"} Oct 30 21:23:00.316919 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:00.316729 5755 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21484dc7dd1d1abef8596e5a8fae3ab36a4fd2295d7ba1bde5a57185e19c2456" Oct 30 21:23:00.316919 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:00.316686 5755 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v5tvk" Oct 30 21:23:11.255177 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:23:11.255135 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:23:19.932910 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:19.932879 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7b5xp_054637a2-b34f-4384-b0af-72d53e95fc94/manager/0.log" Oct 30 21:23:20.531861 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:20.531829 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qhnmd_4267521c-c2c4-4b5e-b10c-b3087da247ac/manager/0.log" Oct 30 21:23:21.414824 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:21.414791 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fmbqh_6d40b739-d433-4eb1-a1de-352637248047/server/0.log" Oct 30 21:23:21.642194 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:21.642163 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-lwcgf_831b546f-889d-4f7a-8bc3-acc64dc3ce83/manager/0.log" Oct 30 21:23:21.965798 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:21.965766 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zb2ww_0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d/proxy/0.log" Oct 30 21:23:23.287483 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:23.287451 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-5ddrt_9c511f0f-6db4-47ac-94e1-c4e831d5f858/revoke-keys/0.log" Oct 30 21:23:23.512212 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:23.512181 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-v5tvk_ba0b956f-3707-44d7-84b0-1b05d3162198/revoke-keys/0.log" Oct 30 21:23:23.737708 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:23.737682 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-s4vtm_007a48c5-b3be-4b1b-83d8-cc5a03364fc3/revoke-keys/0.log" Oct 30 21:23:23.950093 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:23.950064 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-ltwv9_1f2f8b25-b9d5-400a-89d1-f553ad7273e6/postgres/0.log" Oct 30 21:23:26.477051 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:26.477021 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jwmvj_18102e92-59b1-409c-b1cf-fd6b9c1a90b0/kuadrant-console-plugin/0.log" Oct 30 21:23:27.385650 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:27.385620 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-pjpr9_c90843ee-3a81-471e-ba63-4c3ef73eea83/manager/0.log" Oct 30 21:23:28.859260 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:28.859226 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65fc6db558-kxhnz_cbaaf79a-ad29-4b7b-a914-4a2bae26f99e/kube-auth-proxy/0.log" Oct 30 21:23:29.314549 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:29.314520 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-24b4s_cb613fb6-136d-49c9-a4dd-4ac76f320e96/istio-proxy/0.log" Oct 30 21:23:30.070676 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:30.070643 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-l95n7_9ee1db28-bf77-435f-b92f-d6dbd06a5e6a/istio-proxy/0.log" Oct 30 21:23:30.335128 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:30.335055 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-nttzn_e4fdfa05-c965-4c76-a832-fa36a140952e/istio-proxy/0.log" Oct 30 21:23:41.255349 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:23:41.255298 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:23:45.985294 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:45.985259 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7b5xp_054637a2-b34f-4384-b0af-72d53e95fc94/manager/0.log" Oct 30 21:23:46.621169 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:46.621129 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qhnmd_4267521c-c2c4-4b5e-b10c-b3087da247ac/manager/0.log" Oct 30 21:23:47.433401 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:47.433363 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fmbqh_6d40b739-d433-4eb1-a1de-352637248047/server/0.log" Oct 30 21:23:47.719516 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:47.719488 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-lwcgf_831b546f-889d-4f7a-8bc3-acc64dc3ce83/manager/0.log" Oct 30 21:23:48.059420 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:48.059329 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zb2ww_0a1f2d0d-0c5f-45dc-bc49-5b5c112f095d/proxy/0.log" Oct 30 21:23:49.302552 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:49.302515 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-5ddrt_9c511f0f-6db4-47ac-94e1-c4e831d5f858/revoke-keys/0.log" Oct 30 21:23:49.557096 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:49.557014 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-v5tvk_ba0b956f-3707-44d7-84b0-1b05d3162198/revoke-keys/0.log" Oct 30 21:23:49.842208 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:49.842133 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-s4vtm_007a48c5-b3be-4b1b-83d8-cc5a03364fc3/revoke-keys/0.log" Oct 30 21:23:50.114906 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:50.114829 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-ltwv9_1f2f8b25-b9d5-400a-89d1-f553ad7273e6/postgres/0.log" Oct 30 21:23:52.325406 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:52.325363 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jwmvj_18102e92-59b1-409c-b1cf-fd6b9c1a90b0/kuadrant-console-plugin/0.log" Oct 30 21:23:53.312926 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:53.312900 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-pjpr9_c90843ee-3a81-471e-ba63-4c3ef73eea83/manager/0.log" Oct 30 21:23:55.134678 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:55.134645 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65fc6db558-kxhnz_cbaaf79a-ad29-4b7b-a914-4a2bae26f99e/kube-auth-proxy/0.log" Oct 30 21:23:55.726628 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:55.726598 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-24b4s_cb613fb6-136d-49c9-a4dd-4ac76f320e96/istio-proxy/0.log" Oct 30 21:23:56.724471 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:56.724441 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-l95n7_9ee1db28-bf77-435f-b92f-d6dbd06a5e6a/istio-proxy/0.log" Oct 30 21:23:57.025956 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:23:57.025877 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-nttzn_e4fdfa05-c965-4c76-a832-fa36a140952e/istio-proxy/0.log" Oct 30 21:24:11.254814 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:24:11.254778 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:24:41.255126 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:24:41.255048 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:25:11.254728 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:25:11.254694 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:25:18.748341 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:18.748309 5755 scope.go:122] "RemoveContainer" containerID="5dc4d5177087a1ecb5464981dfd13a4540ff30ee7f879ec0a5f6cba4573d73aa" Oct 30 21:25:24.576000 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:24.575963 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lwk8k_95f11ca1-5d71-4135-bf16-0fe6c24706c0/global-pull-secret-syncer/0.log" Oct 30 21:25:27.498476 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:27.498447 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6kfsl_dff6e046-945d-4553-ba94-908dbb1a0c23/konnectivity-agent/0.log" Oct 30 21:25:29.230457 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:29.230426 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-195.us-west-1.compute.internal_b1b5f93e01c03391b49a05418cff0a8e/haproxy/0.log" Oct 30 21:25:33.444328 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.444296 5755 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc"] Oct 30 21:25:33.444745 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.444720 5755 cpu_manager.go:414] "containerMap: removing container" podUID="ba0b956f-3707-44d7-84b0-1b05d3162198" containerName="revoke-keys" Oct 30 21:25:33.444745 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.444733 5755 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0b956f-3707-44d7-84b0-1b05d3162198" containerName="revoke-keys" Oct 30 21:25:33.444823 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.444794 5755 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0b956f-3707-44d7-84b0-1b05d3162198" containerName="revoke-keys" Oct 30 21:25:33.449213 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.449197 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:33.480130 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.480094 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce3b0dce-a409-48ba-9969-30e80818177c-podres\") pod \"perf-node-gather-daemonset-hvscc\" (UID: \"ce3b0dce-a409-48ba-9969-30e80818177c\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:33.480130 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.480132 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce3b0dce-a409-48ba-9969-30e80818177c-proc\") pod \"perf-node-gather-daemonset-hvscc\" (UID: \"ce3b0dce-a409-48ba-9969-30e80818177c\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:33.480328 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.480260 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4xdp\" (UniqueName: \"kubernetes.io/projected/ce3b0dce-a409-48ba-9969-30e80818177c-kube-api-access-r4xdp\") pod \"perf-node-gather-daemonset-hvscc\" (UID: \"ce3b0dce-a409-48ba-9969-30e80818177c\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:33.480380 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.480364 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce3b0dce-a409-48ba-9969-30e80818177c-lib-modules\") pod \"perf-node-gather-daemonset-hvscc\" (UID: \"ce3b0dce-a409-48ba-9969-30e80818177c\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:33.480460 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.480420 5755 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce3b0dce-a409-48ba-9969-30e80818177c-sys\") pod \"perf-node-gather-daemonset-hvscc\" (UID: \"ce3b0dce-a409-48ba-9969-30e80818177c\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:33.501921 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.501892 5755 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x4r57\"/\"default-dockercfg-sq655\"" Oct 30 21:25:33.551490 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.551452 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x4r57\"/\"openshift-service-ca.crt\"" Oct 30 21:25:33.551643 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.551508 5755 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x4r57\"/\"kube-root-ca.crt\"" Oct 30 21:25:33.553505 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.553482 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc"] Oct 30 21:25:33.581876 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.581847 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce3b0dce-a409-48ba-9969-30e80818177c-podres\") pod \"perf-node-gather-daemonset-hvscc\" (UID: \"ce3b0dce-a409-48ba-9969-30e80818177c\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:33.581876 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.581876 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce3b0dce-a409-48ba-9969-30e80818177c-proc\") pod \"perf-node-gather-daemonset-hvscc\" (UID: \"ce3b0dce-a409-48ba-9969-30e80818177c\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:33.582099 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.581915 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4xdp\" (UniqueName: \"kubernetes.io/projected/ce3b0dce-a409-48ba-9969-30e80818177c-kube-api-access-r4xdp\") pod \"perf-node-gather-daemonset-hvscc\" (UID: \"ce3b0dce-a409-48ba-9969-30e80818177c\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:33.582099 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.581948 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce3b0dce-a409-48ba-9969-30e80818177c-lib-modules\") pod \"perf-node-gather-daemonset-hvscc\" (UID: \"ce3b0dce-a409-48ba-9969-30e80818177c\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:33.582099 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.581975 5755 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce3b0dce-a409-48ba-9969-30e80818177c-sys\") pod \"perf-node-gather-daemonset-hvscc\" (UID: \"ce3b0dce-a409-48ba-9969-30e80818177c\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:33.582099 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.582018 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce3b0dce-a409-48ba-9969-30e80818177c-lib-modules\") pod \"perf-node-gather-daemonset-hvscc\" (UID: \"ce3b0dce-a409-48ba-9969-30e80818177c\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:33.582099 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.582025 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce3b0dce-a409-48ba-9969-30e80818177c-proc\") pod \"perf-node-gather-daemonset-hvscc\" (UID: \"ce3b0dce-a409-48ba-9969-30e80818177c\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:33.582099 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.581972 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce3b0dce-a409-48ba-9969-30e80818177c-podres\") pod \"perf-node-gather-daemonset-hvscc\" (UID: \"ce3b0dce-a409-48ba-9969-30e80818177c\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:33.582099 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.582057 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce3b0dce-a409-48ba-9969-30e80818177c-sys\") pod \"perf-node-gather-daemonset-hvscc\" (UID: \"ce3b0dce-a409-48ba-9969-30e80818177c\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:33.640836 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.640799 5755 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4xdp\" (UniqueName: \"kubernetes.io/projected/ce3b0dce-a409-48ba-9969-30e80818177c-kube-api-access-r4xdp\") pod \"perf-node-gather-daemonset-hvscc\" (UID: \"ce3b0dce-a409-48ba-9969-30e80818177c\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:33.762037 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:33.762008 5755 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:34.136344 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:34.136312 5755 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc"] Oct 30 21:25:34.138618 ip-10-0-139-195 kubenswrapper[5755]: W1030 21:25:34.138590 5755 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podce3b0dce_a409_48ba_9969_30e80818177c.slice/crio-ee1c63ffd8ae133872d8d8307e7aad98240af2a2483bad4f9a577054f49a7ce2 WatchSource:0}: Error finding container ee1c63ffd8ae133872d8d8307e7aad98240af2a2483bad4f9a577054f49a7ce2: Status 404 returned error can't find the container with id ee1c63ffd8ae133872d8d8307e7aad98240af2a2483bad4f9a577054f49a7ce2 Oct 30 21:25:34.403488 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:34.403414 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jwmvj_18102e92-59b1-409c-b1cf-fd6b9c1a90b0/kuadrant-console-plugin/0.log" Oct 30 21:25:35.087973 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:35.087941 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" event={"ID":"ce3b0dce-a409-48ba-9969-30e80818177c","Type":"ContainerStarted","Data":"eea4210e54650909df7067311b6c1abcc425898eebe80b1af3611dab85d7666f"} Oct 30 21:25:35.088513 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:35.087992 5755 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" event={"ID":"ce3b0dce-a409-48ba-9969-30e80818177c","Type":"ContainerStarted","Data":"ee1c63ffd8ae133872d8d8307e7aad98240af2a2483bad4f9a577054f49a7ce2"} Oct 30 21:25:35.088513 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:35.088067 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:35.202569 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:35.202515 5755 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" podStartSLOduration=2.202502733 podStartE2EDuration="2.202502733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:35.200919015 +0000 UTC m=+3498.127475929" watchObservedRunningTime="2026-10-30 21:25:35.202502733 +0000 UTC m=+3498.129059647" Oct 30 21:25:35.808791 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:35.808755 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-pjpr9_c90843ee-3a81-471e-ba63-4c3ef73eea83/manager/0.log" Oct 30 21:25:41.102767 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:41.102736 5755 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-hvscc" Oct 30 21:25:41.257322 ip-10-0-139-195 kubenswrapper[5755]: E1030 21:25:41.257288 5755 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hkvc" containerName="extractor" Oct 30 21:25:41.934606 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:41.934570 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ffxbf_f019c85d-f532-4848-905b-b7e4480adba1/kube-state-metrics/0.log" Oct 30 21:25:42.384786 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:42.384726 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ffxbf_f019c85d-f532-4848-905b-b7e4480adba1/kube-rbac-proxy-main/0.log" Oct 30 21:25:42.809101 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:42.809052 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ffxbf_f019c85d-f532-4848-905b-b7e4480adba1/kube-rbac-proxy-self/0.log" Oct 30 21:25:46.650532 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:46.650497 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qm6mr_420b750b-2a29-4432-84f7-0d7709493a41/node-exporter/0.log" Oct 30 21:25:47.072980 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:47.072951 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qm6mr_420b750b-2a29-4432-84f7-0d7709493a41/kube-rbac-proxy/0.log" Oct 30 21:25:47.496535 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:47.496510 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qm6mr_420b750b-2a29-4432-84f7-0d7709493a41/init-textfile/0.log" Oct 30 21:25:47.923755 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:47.923681 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-tpsmf_3eb20209-5da1-4ef7-8625-b06c2b6493ef/kube-rbac-proxy-main/0.log" Oct 30 21:25:48.347730 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:48.347701 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-tpsmf_3eb20209-5da1-4ef7-8625-b06c2b6493ef/kube-rbac-proxy-self/0.log" Oct 30 21:25:48.774811 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:48.774786 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-tpsmf_3eb20209-5da1-4ef7-8625-b06c2b6493ef/openshift-state-metrics/0.log" Oct 30 21:25:49.214501 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:49.214417 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c2d68c75-b92b-419b-b408-a0e0799e7f72/prometheus/0.log" Oct 30 21:25:49.639683 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:49.639654 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c2d68c75-b92b-419b-b408-a0e0799e7f72/config-reloader/0.log" Oct 30 21:25:50.062142 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:50.062109 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c2d68c75-b92b-419b-b408-a0e0799e7f72/thanos-sidecar/0.log" Oct 30 21:25:50.485806 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:50.485779 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c2d68c75-b92b-419b-b408-a0e0799e7f72/kube-rbac-proxy-web/0.log" Oct 30 21:25:50.910423 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:50.910335 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c2d68c75-b92b-419b-b408-a0e0799e7f72/kube-rbac-proxy/0.log" Oct 30 21:25:51.342808 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:51.342775 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c2d68c75-b92b-419b-b408-a0e0799e7f72/kube-rbac-proxy-thanos/0.log" Oct 30 21:25:51.766761 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:51.766731 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c2d68c75-b92b-419b-b408-a0e0799e7f72/init-config-reloader/0.log" Oct 30 21:25:53.045132 ip-10-0-139-195 kubenswrapper[5755]: I1030 21:25:53.045101 5755 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-5kk8q_3b1f827c-2d6d-4d1d-ac35-23324ff897fa/prometheus-operator-admission-webhook/0.log"